Lesson 35: ROC, PR Curves & Calibration

1. ROC, PR Curves & Calibration

Title

USAAIO · Lesson 35 · Week 12 (Evaluation)

Threshold-free evaluation, and probabilities you can trust. We build the ROC curve one threshold at a time, compute its AUC three independent ways, expose why it lies under imbalance, then fix an overconfident model with temperature scaling — every number produced by real execution.

2. By the end of this lesson you can

Objectives

1. Read a confusion matrix at a threshold and compute TPR, FPR, and precision from it by hand
2. Build the ROC curve point by point by sweeping the threshold, and integrate it for the AUC
3. Prove AUC = P(random positive scores above random negative) by counting concordant pairs — the same 0.875 three ways
4. Build the precision-recall curve and average precision, and say exactly why ROC-AUC flatters an imbalanced model while PR-AUC doesn't
5. Define calibration and the Expected Calibration Error, and drive ECE to its minimum with temperature scaling T

3. What survived from Concentration Inequalities & Generalization?

Warm-up

4. The confusion matrix

Section

Part 1 of 6 — the foundation

5. One threshold turns scores into decisions

Concept

A classifier outputs a score s ∈ [0,1] — how positive it thinks each example is. To make an actual decision you pick a threshold t and call everything with s ≥ t positive.

Change t and the decisions change. Evaluation metrics like accuracy depend on that one arbitrary knob — the whole point of ROC and PR curves is to escape it by sweeping t across all its values at once.

6. Break it if you can: One threshold turns scores into decisions

Counterexample

7. The data: ten scored examples

Concept

One running example for the whole lesson. Ten examples, each with a model score s and a true label y (1 = positive). Sorted high score to low:

There are P = 4 positives and N = 6 negatives. Notice the ranking is good but not perfect: example 3 (a negative) scores 0.80, above example 4 (a positive) at 0.70. That one swap is what keeps the AUC below 1.

8. Watch it run: The data: ten scored examples

Pattern

9. The four confusion-matrix counts

Concept

Fix a threshold. Every example falls into exactly one of four boxes, comparing what the model predicted against the truth.

TP / FP / FN / TN — TP: called positive, was positive (a hit). FP: called positive, was negative (a false alarm). FN: called negative, was positive (a miss). TN: called negative, was negative (a correct rejection). Every metric today is built from just these four numbers.

10. Fill in: predicted − for The four confusion-matrix counts

Comparison

11. Guess the shape of the answer: Count the four boxes at t = 0.5

Estimation

12. Count the four boxes at t = 0.5

Worked example

Threshold t = 0.5: predict positive when s ≥ 0.5. That flags examples 1–5 (scores 0.95, 0.90, 0.80, 0.70, 0.55). Compare each flag to its true label:

\[ TP = 3, \quad FP = 2, \quad FN = 1, \quad TN = 4 \qquad (TP+FP+FN+TN = 10\;\checkmark) \]

13. Decode the notation: Count the four boxes at t = 0.5

Notation

14. Three rates read off the boxes

Concept

The two axes of the ROC curve, plus precision, are ratios of those four counts. Learn which denominator each one uses — that choice is the whole story of imbalance.

\[ \text{TPR (recall)} = \frac{TP}{TP+FN}, \qquad \text{FPR} = \frac{FP}{FP+TN}, \qquad \text{precision} = \frac{TP}{TP+FP} \]

TPR (recall) is the fraction of real positives caught. FPR is the fraction of real negatives falsely flagged. Precision is the fraction of flags that were right. TPR and FPR split by the true class; precision splits by the predicted class.

15. By analogy: Three rates read off the boxes

Analogy

16. What has to be given first: The rates at t = 0.5, verified in code

Missing information

17. The rates at t = 0.5, verified in code

Worked example

Plug TP=3, FP=2, FN=1, TN=4 into the three formulas, then confirm with a runnable snippet that rebuilds the whole thing from the raw scores:

import numpy as np
scores = np.array([0.95,0.90,0.80,0.70,0.55,0.45,0.40,0.30,0.20,0.10])
y      = np.array([1,   1,   0,   1,   0,   1,   0,   0,   0,   0  ])
thr = 0.5
pred = (scores >= thr).astype(int)
TP = int(((pred==1)&(y==1)).sum())
FP = int(((pred==1)&(y==0)).sum())
FN = int(((pred==0)&(y==1)).sum())
TN = int(((pred==0)&(y==0)).sum())
print("TP FP FN TN =", TP, FP, FN, TN)
print("TPR =", TP/(TP+FN))
print("FPR =", FP/(FP+TN))
print("precision =", TP/(TP+FP))

18. What each one costs: The rates at t = 0.5, verified in code

Trade off

19. The ROC curve

Section

Part 2 of 6 — sweep every threshold

20. Why one point is not enough

Intuition

The (0.333, 0.75) point above depended entirely on our arbitrary t = 0.5. A different threshold gives a different trade-off — lower t catches more positives (higher TPR) but raises false alarms (higher FPR).

Rather than defend one threshold, plot all of them. Slide t from +∞ (nothing flagged) down to −∞ (everything flagged) and trace the (FPR, TPR) point as it moves.

That trace is the ROC curve — Receiver Operating Characteristic. It shows the model's entire achievable frontier of trade-offs in one picture, independent of any single threshold choice.

21. Teach it back: Why one point is not enough

Explain it

22. How the curve moves as t drops

Concept

Start with t above every score: nothing is flagged, so TP = FP = 0 and the point sits at the origin (0, 0). Now lower t past one example's score at a time.

Each example you cross adds one flag. If that example is a real positive, TP grows → the point steps up. If it is a real negative, FP grows → the point steps right.

So the ROC curve is a staircase from (0,0) to (1,1): up on a positive, right on a negative. A good model front-loads all its ups (climbs the left edge early) before the rights.

23. Predict the next row: Build the ROC curve from scratch

Pattern

24. Build the ROC curve from scratch

Worked example

Sort by descending score, walk the examples, and after each one record the running (FPR, TPR). This snippet is complete and self-contained — it prints every point on the curve:

import numpy as np
from sklearn.metrics import roc_auc_score
scores = np.array([0.95,0.90,0.80,0.70,0.55,0.45,0.40,0.30,0.20,0.10])
y      = np.array([1,   1,   0,   1,   0,   1,   0,   0,   0,   0  ])
P, N = y.sum(), (1-y).sum()
tp = fp = 0
tpr = [0.0]; fpr = [0.0]
for i in np.argsort(-scores):
    tp += y[i]; fp += 1 - y[i]
    tpr.append(tp / P); fpr.append(fp / N)
print(round(np.trapz(tpr, fpr), 4), round(roc_auc_score(y, scores), 4))

25. Reading the staircase

Concept

Trace the table: the first two crossings are positives, so the point climbs straight up the left edge to TPR = 0.5 with FPR still 0. Then example 3 (score 0.80, a negative) forces the first step right.

After example 6 (the last positive, score 0.45), TPR hits 1.0 — every positive is caught. The remaining four crossings are all negatives, so the curve runs flat across the top from FPR = 0.333 to 1.0.

26. AUC, three ways

Section

Part 2 continued — 0.875 every time

27. AUC: the area under the curve

Concept

One number summarizes the whole staircase: the area under the ROC curve, AUC ∈ [0,1]. A curve that hugs the top-left corner encloses nearly the full unit square (AUC → 1); the random diagonal encloses half (AUC = 0.5).

For a staircase, the area is a sum of rectangles: at each step right (width ΔFPR), the height is the current TPR. That is exactly what the trapezoid rule np.trapz(tpr, fpr) computes.

28. What has to happen first: Way 1 — trapezoid area by hand

Ranking

29. Way 1 — trapezoid area by hand

Worked example

The curve only advances rightward at the four negative crossings. Sum TPR × ΔFPR over just those steps (each ΔFPR = 1/6). Read the TPR at each right-step from the table:

\[ \text{AUC} = \tfrac{1}{6}\big(0.5 + 0.75 + 1.0 + 1.0 + 1.0 + 1.0\big) = \tfrac{1}{6}(5.25) = 0.875 \]

30. Work backwards from the answer: Way 1 — trapezoid area by hand

Reverse engineer

31. Restore the missing line: Way 2 — np.trapz matches sklearn

Fill the middle

32. Way 2 — np.trapz matches sklearn

Worked example

The same block from before ends by integrating the curve and printing it beside sklearn's roc_auc_score. Both must agree with our hand value 0.875:

import numpy as np
from sklearn.metrics import roc_auc_score
scores = np.array([0.95,0.90,0.80,0.70,0.55,0.45,0.40,0.30,0.20,0.10])
y      = np.array([1,   1,   0,   1,   0,   1,   0,   0,   0,   0  ])
P, N = y.sum(), (1-y).sum()
tp = fp = 0; tpr = [0.0]; fpr = [0.0]
for i in np.argsort(-scores):
    tp += y[i]; fp += 1 - y[i]
    tpr.append(tp / P); fpr.append(fp / N)
auc_manual = np.trapz(tpr, fpr)
print(round(auc_manual, 4), round(roc_auc_score(y, scores), 4))

33. Where does each piece belong: Lesson 35: ROC, PR Curves & Calibration

Sorting

34. AUC is a ranking probability

Intuition

Here is the interpretation worth memorizing: AUC is the probability that a randomly chosen positive scores higher than a randomly chosen negative.

That is why AUC is threshold-free — it never fixes a t, it only asks whether the model ranks positives above negatives. A perfect ranker always does (AUC = 1); a coin flip does it half the time (0.5).

\[ \text{AUC} = \Pr\big(\,s^{+} > s^{-}\,\big) \quad\text{over random }(s^{+},s^{-})\text{ pairs} \]

35. Way 3 — count concordant pairs

Worked example

Make that probability literal: form every (positive, negative) pair, count how many the positive outranks, and divide by the total P·N = 4·6 = 24:

import numpy as np
scores = np.array([0.95,0.90,0.80,0.70,0.55,0.45,0.40,0.30,0.20,0.10])
y      = np.array([1,   1,   0,   1,   0,   1,   0,   0,   0,   0  ])
pos = scores[y==1]; neg = scores[y==0]
concordant = 0.0
for sp in pos:
    for sn in neg:
        if sp > sn:    concordant += 1.0
        elif sp == sn: concordant += 0.5
print(concordant, len(pos)*len(neg))
print(round(concordant/(len(pos)*len(neg)), 4))

36. Inspect it line by line: Way 3 — count concordant pairs

Error analysis

37. Three reference curves

Concept

Memorize the three landmarks so you can read any AUC instantly:

A worse-than-random model still has signal — it just points backwards. Flip its predictions and the AUC becomes 1 − AUC > 0.5. Never discard an AUC of 0.3.

38. Restore the missing line: Flipping a backwards classifier

Fill the middle

39. Flipping a backwards classifier

Worked example

Same scores, but now the labels are inverted so the model ranks positives too low. Its AUC lands below 0.5 — until we flip the scores:

import numpy as np
from sklearn.metrics import roc_auc_score
scores = np.array([0.95,0.90,0.80,0.70,0.55,0.45,0.40,0.30,0.20,0.10])
y      = np.array([0,   0,   1,   0,   1,   0,   1,   1,   1,   1  ])
auc         = roc_auc_score(y, scores)
auc_flipped = roc_auc_score(y, 1 - scores)
print(round(auc, 3), round(auc_flipped, 3), round(1 - auc, 3))

40. Work backwards from the answer: Flipping a backwards classifier

Reverse engineer

41. Something is wrong here: swapping the ROC axes

Anomaly

42. Trap: swapping the ROC axes

Trap

43. Decode the notation: Trap: swapping the ROC axes

Notation

44. The PR curve & imbalance

Section

Part 3 of 6 — where ROC lies

45. Precision-recall: focus on the positives

Concept

The precision-recall curve plots precision (TP/(TP+FP)) on the y-axis against recall (= TPR = TP/(TP+FN)) on the x-axis, swept over the same thresholds.

Its summary number is the average precision (AP), the PR-curve analogue of AUC — the area under the precision-recall curve.

The key difference from ROC: neither precision nor recall uses TN. PR ignores the true negatives entirely and looks only at the positive class — exactly the class you care about when positives are rare.

46. Guess the shape of the answer: Build the PR curve and AP from scratch

Estimation

47. Build the PR curve and AP from scratch

Worked example

Same descending sweep, but now track precision and recall, and accumulate AP as Σ precision·Δrecall at each positive crossing:

import numpy as np
from sklearn.metrics import average_precision_score
scores = np.array([0.95,0.90,0.80,0.70,0.55,0.45,0.40,0.30,0.20,0.10])
y      = np.array([1,   1,   0,   1,   0,   1,   0,   0,   0,   0  ])
P = y.sum(); tp = fp = 0; ap = 0.0; prev_rec = 0.0
for i in np.argsort(-scores):
    tp += y[i]; fp += 1 - y[i]
    prec = tp / (tp + fp); rec = tp / P
    if y[i] == 1:
        ap += prec * (rec - prev_rec); prev_rec = rec
print(round(ap, 4), round(average_precision_score(y, scores), 4))

48. Which is which, by label

Discrimination

49. AP matches sklearn exactly

Concept

Our hand-accumulated AP is 0.8542, and average_precision_score returns the identical 0.8542. The manual PR curve is faithful.

On this balanced-ish data (4/10 positive) AP 0.854 and AUC 0.875 are close. The gap explodes only when positives become rare — which is the next experiment.

50. F1: one number at one threshold

Concept

PR-AUC summarizes the whole curve. If instead you must pick one threshold, the F1 score balances precision and recall at that operating point via their harmonic mean:

\[ F_1 = \frac{2 \cdot \text{precision} \cdot \text{recall}}{\text{precision} + \text{recall}} \]

The harmonic mean punishes imbalance between the two: F1 is high only when precision and recall are both high. A model with precision 1.0 but recall 0.1 scores F1 ≈ 0.18, not 0.55 — one weak side drags it down.

51. Why harmonic, not arithmetic, mean

Intuition

Averaging precision and recall the ordinary way would let a lopsided model cheat: precision 1.0, recall 0.1 averages to 0.55 — sounds decent, but the model finds only one positive in ten.

The harmonic mean is dominated by the smaller number. It sits near the weaker of the two, so you can't paper over a terrible recall with a great precision. That is exactly the behavior you want from a single 'is this model actually usable' score.

Same reason a car that goes 100 mph one way and 1 mph back doesn't average 50 mph over the trip — time is dominated by the slow leg. F1 is dominated by the weak metric.

52. Finish it with less help: Pick the operating threshold

Faded example

53. Pick the operating threshold

Worked example

PR-AUC and F1 don't tell you which threshold to deploy. Sweep them, then choose by your requirement: best F1, or the most recall you can get while holding precision ≥ 0.75. Runnable and self-contained:

import numpy as np
scores = np.array([0.95,0.90,0.80,0.70,0.55,0.45,0.40,0.30,0.20,0.10])
y      = np.array([1,   1,   0,   1,   0,   1,   0,   0,   0,   0  ])
P = y.sum(); tp = fp = 0; rows = []
for i in np.argsort(-scores):
    tp += y[i]; fp += 1 - y[i]
    prec = tp/(tp+fp); rec = tp/P
    f1 = 0.0 if prec+rec == 0 else 2*prec*rec/(prec+rec)
    rows.append((scores[i], prec, rec, f1))
best = max(rows, key=lambda r: r[3])
print("best F1", round(best[3],3), "at thr", best[0])

54. Watch it run: Pick the operating threshold

Pattern

55. Why ROC-AUC flatters rare positives

Intuition

FPR's denominator is FP + TN — the count of all negatives. When 98% of the data is negative, that denominator is enormous, so even hundreds of false positives barely move FPR off zero.

So the ROC curve stays pinned to the left edge and the AUC looks great — while in reality the model floods you with false alarms for every true positive it finds.

Precision's denominator is TP + FP — only the flagged examples. It has no giant TN to hide behind, so it feels every false positive. That is why PR is the honest metric under imbalance.

56. Restore the missing line: ROC-AUC vs PR-AUC under 2% imbalance

Fill the middle

57. ROC-AUC vs PR-AUC under 2% imbalance

Worked example

Train logistic regression on a synthetic dataset that is only 2% positive, then compare the two summary numbers on held-out data. Fully runnable and seeded:

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score, average_precision_score
X, y = make_classification(n_samples=4000, weights=[0.98, 0.02],
                           n_informative=5, random_state=1)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.5, random_state=0)
p = LogisticRegression(max_iter=2000).fit(Xtr, ytr).predict_proba(Xte)[:, 1]
print(round(yte.mean(), 4))
print(round(roc_auc_score(yte, p), 3), round(average_precision_score(yte, p), 3))

58. Fill in: story for ROC-AUC vs PR-AUC under 2% imbalance

Comparison

59. Something is wrong here: judging imbalance by ROC-AUC

Anomaly

60. Trap: judging imbalance by ROC-AUC

Trap

61. Break it on purpose: judging imbalance by ROC-AUC

Break the constraint

62. Rebuild the recipe: The curve-selection recipe

Ranking

63. The curve-selection recipe

Pattern

1. Build ROC: sweep threshold high→low; step up on a positive, right on a negative; area = AUC
2. Read AUC as P(random positive > random negative) — threshold-free ranking quality
3. Reference shapes: perfect 1.0, random 0.5, worse < 0.5 (flip it → 1 − AUC)
4. Balanced-ish classes? ROC-AUC is fine
5. Rare positives? Switch to PR-AUC / average precision, and compare it to the prevalence baseline, never ROC-AUC

64. Where does it stop working: The curve-selection recipe

Edge cases

65. Rule out three: Check yourself — worse than random

Elimination

66. Check yourself — worse than random

Check

Read the AUC before you answer.

67. Answer it before you see the options: Check yourself — which curve under…

Prediction

68. Check yourself — which curve under imbalance

Check

Think about which denominator hides the false alarms.

69. Calibration

Section

Part 4 of 6 — trust the probabilities

70. Ranking is not the same as honesty

Intuition

AUC and AP only judge ranking — do positives score above negatives. They say nothing about whether a 0.9 really means a 90% chance.

A model can rank perfectly (AUC = 1) yet output 0.9 on cases that are right only 60% of the time. Its scores are ordered correctly but dishonest as probabilities.

Whenever a downstream decision uses the probability itself — expected cost, a betting threshold, medical risk — you need the scores to be calibrated, not just well-ranked.

71. What calibration means

Concept

A model is calibrated if its confidence matches reality: among all the cases where it predicts 0.8, about 80% should actually be positive.

reliability diagram — A plot of predicted confidence (x) against observed accuracy (y), one point per confidence bin. Perfect calibration lies on the diagonal. Points BELOW the diagonal mean the model is overconfident — its confidence exceeds its accuracy — which is the typical failure of modern deep nets.

72. Take the definitions apart: TP / FP / FN / TN vs reliability diagram

Definition probe

73. Discrimination and calibration are independent

Concept

These are two separate axes of a model, and a model can be good on one and bad on the other:

Our ×3-overconfident model lives in the top-right box: its ranking is untouched (temperature scaling proves the argmax never moves), yet its probabilities are dishonest. Fixing calibration leaves AUC exactly where it was — the two axes don't interfere.

74. Expected Calibration Error (ECE)

Concept

ECE turns the reliability gap into one number. Bin the predictions by confidence, and in each bin take the absolute gap between mean confidence and observed accuracy, weighted by how many points fall in the bin:

\[ \text{ECE} = \sum_{b=1}^{B} \frac{n_b}{n}\,\big|\,\text{acc}_b - \text{conf}_b\,\big| \]

accᵦ is the fraction of bin b that is actually positive; confᵦ is the average predicted probability in that bin; nᵦ/n weights each bin by its share of the data. ECE = 0 is perfect calibration.

75. Predict the next row: ECE bin by bin on an overconfident model

Pattern

76. ECE bin by bin on an overconfident model

Worked example

Make the abstraction concrete. Take the ×3-overconfident model, split its predictions into 10 confidence bins, and print each bin's count, mean confidence, and observed accuracy. Self-contained and seeded:

import numpy as np
np.random.seed(0)
sig = lambda z: 1.0/(1.0+np.exp(-z))
z = np.random.randn(4000)*1.5
labels = (np.random.rand(4000) < sig(z)).astype(float)
probs = sig(z*3.0)                 # overconfident predictions
edges = np.linspace(0, 1, 11); ece = 0.0
for i in range(10):
    m = (probs > edges[i]) & (probs <= edges[i+1])
    if m.sum():
        conf, acc = probs[m].mean(), labels[m].mean()
        ece += m.mean()*abs(acc - conf)
        print(f"{edges[i]:.1f}-{edges[i+1]:.1f} n={m.sum():4d} conf={conf:.3f} acc={acc:.3f}")
print("ECE =", round(ece, 4))

77. Watch it run: ECE bin by bin on an overconfident model

Pattern

78. Three post-hoc fixes

Concept

Calibration is repaired after training, without touching the model's ranking. Three standard tools:

• Platt scaling — fit a sigmoid σ(a·s + b) to map scores to calibrated probabilities (two parameters).
• Isotonic regression — fit any monotone step function; more flexible, needs more data.
• Temperature scaling — divide the logits by a single learned T > 0 before the sigmoid/softmax. One parameter, cannot change the ranking.

Temperature scaling is the workhorse for neural nets: one number, provably preserves accuracy, and directly attacks overconfidence. We derive and sweep it next.

79. The temperature mechanism

Concept

A model's raw pre-sigmoid output is its logit z. The probability is p = σ(z) = 1/(1 + e^{−z}). Temperature scaling divides the logit by T before the sigmoid:

\[ p_T = \sigma\!\left(\frac{z}{T}\right) = \frac{1}{1 + e^{-z/T}} \]

T > 1 shrinks the logits toward 0, pulling probabilities toward 0.5 — it softens an overconfident model. T < 1 sharpens them. T = 1 leaves the model unchanged.

80. Teach it back: The temperature mechanism

Explain it

81. What has to be given first: Temperature scaling drives ECE down

Missing information

82. Temperature scaling drives ECE down

Worked example

Build a well-calibrated latent (label ~ Bernoulli(σ(z))), then make the model overconfident by reporting 3z instead of z. Sweep T and watch ECE. Fully self-contained and seeded:

import numpy as np
np.random.seed(0)
sig = lambda z: 1.0/(1.0+np.exp(-z))
def ece(probs, labels, bins=10):
    e = 0.0; edges = np.linspace(0, 1, bins+1)
    for i in range(bins):
        m = (probs > edges[i]) & (probs <= edges[i+1])
        if m.sum():
            e += m.mean()*abs(probs[m].mean() - labels[m].mean())
    return e
n = 4000
z = np.random.randn(n)*1.5
labels = (np.random.rand(n) < sig(z)).astype(float)
logits = z*3.0                      # overconfident model
for T in [1, 2, 3, 4, 5]:
    print(T, round(ece(sig(logits/T), labels), 4))

83. Inspect it line by line: Temperature scaling drives ECE down

Error analysis

84. Reading the ECE sweep

Concept

ECE falls from 0.1448 at T = 1 to its minimum 0.0176 at T = 3, then climbs back to 0.0830 at T = 5. The curve is a valley: too little softening leaves overconfidence, too much creates underconfidence.

The minimum sits exactly at T = 3 because that is the factor by which we inflated the logits. In practice you don't know the true factor — you fit T by minimizing ECE (or NLL) on a held-out set.

85. Guess the shape of the answer: Temperature never changes the prediction

Estimation

86. Temperature never changes the prediction

Worked example

Crucial property: dividing all logits by the same positive T preserves their order, so the argmax — the actual prediction — never moves. Accuracy is untouched:

import numpy as np
sig = lambda z: 1.0/(1.0+np.exp(-z))
logits = np.array([2.0, -1.0, 0.5, 3.0, -2.5])
p1 = sig(logits/1.0)
p3 = sig(logits/3.0)
print(np.round(p1, 3))
print(np.round(p3, 3))
print((np.argsort(-p1) == np.argsort(-p3)).all())
print((p1 > 0.5).astype(int), (p3 > 0.5).astype(int))

87. Which is which, by pred (both)

Discrimination

88. Something is wrong here: confident means correct

Anomaly

89. Trap: confident means correct

Trap

90. Which of these survive contact with Lesson 35: ROC, PR Curves & Calibration?

Two truths and a lie

91. One knob, two jobs

Section

Part 5 of 6 — temperature elsewhere

92. Temperature in LLM sampling

Concept

The exact same T knob controls text generation. Before the softmax over next-token logits, dividing by T reshapes the sampling distribution — same mechanism, different goal.

\[ \Pr(\text{token}_k) = \frac{e^{z_k / T}}{\sum_j e^{z_j / T}} \]

T < 1 sharpens toward the top token (more deterministic, 'greedy'); T > 1 flattens toward uniform (more diverse, more random). T → 0 is pure argmax; T → ∞ is a coin flip over the vocabulary.

93. By analogy: Temperature in LLM sampling

Analogy

94. Same math, two purposes

Intuition

In calibration, you fit T to a fixed value that makes probabilities honest — you want the number to be trustworthy.

In sampling, you choose T as a creativity dial — low for factual answers, high for brainstorming. Nothing is being calibrated; you're tuning randomness.

Recognizing that σ(z/T) and softmax(z/T) are the same operation is the kind of connection USAAIO loves to test — calibration (this lesson) and softmax temperature (Lesson 17) are one idea.

95. Break it if you can: Same math, two purposes

Counterexample

96. Answer it before you see the options: Check yourself — what temperature does

Prediction

97. Check yourself — what temperature does

Check

T > 1 applied to logits — think about the argmax.

98. Rule out three: Check yourself — reading an AUC point

Elimination

99. Check yourself — reading an AUC point

Check

Recall what a single ROC point encodes.

100. Your turn: build it

Section

Part 6 of 6 — the project

101. Project: ROC from scratch → PR → calibration

Concept

Assemble the whole lesson yourself: build an ROC curve and AUC by hand, contrast ROC-AUC with PR-AUC on imbalanced data, then recalibrate an overconfident model with temperature scaling. You derived every piece — now wire them together.

Build rules: type every line yourself, sort by descending score for the ROC sweep, and confirm your manual AUC matches sklearn before moving on. When a shape error hits, print .shape — read the error, don't delete it.

102. Milestone 1 — ROC & AUC by hand

Worked example

Your turn: sweep thresholds to accumulate TPR/FPR, integrate for the AUC, and check it against sklearn. Predict out loud: will they match exactly?

Hint: walk np.argsort(-scores) (descending). Each example adds to tp if positive or fp if negative; append tp/P and fp/N; finish with np.trapz(tpr, fpr).

import numpy as np
from sklearn.metrics import roc_auc_score
scores = np.array([0.95,0.90,0.80,0.70,0.55,0.45,0.40,0.30,0.20,0.10])
y      = np.array([1,   1,   0,   1,   0,   1,   0,   0,   0,   0  ])
P, N = y.sum(), (1-y).sum()
tp = fp = 0; tpr = [0.0]; fpr = [0.0]
for i in np.argsort(-scores):
    tp += y[i]; fp += 1 - y[i]
    tpr.append(tp/P); fpr.append(fp/N)
print(round(np.trapz(tpr, fpr), 4), round(roc_auc_score(y, scores), 4))

103. Milestone 2 — ROC-AUC vs PR-AUC

Worked example

Your turn: train logistic regression on 2%-positive data and print both AUCs. Predict which one collapses.

Hint: make_classification(weights=[0.98, 0.02], ...), split, predict_proba(...)[:, 1], then roc_auc_score vs average_precision_score. Keep the same random_states to reproduce the numbers.

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score, average_precision_score
X, y = make_classification(n_samples=4000, weights=[0.98, 0.02],
                           n_informative=5, random_state=1)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.5, random_state=0)
p = LogisticRegression(max_iter=2000).fit(Xtr, ytr).predict_proba(Xte)[:, 1]
print(round(roc_auc_score(yte, p), 3), round(average_precision_score(yte, p), 3))

104. Milestone 3 — temperature scaling

Worked example

Your turn: write an ece() function, build an overconfident model (×3 logits), and sweep T. Predict the best T before you run it.

Hint: ece bins predictions into 10 bins and averages |conf − acc| weighted by bin size. The overconfidence factor is 3, so the ECE minimum should land at T ≈ 3.

import numpy as np
np.random.seed(0)
sig = lambda z: 1.0/(1.0+np.exp(-z))
def ece(probs, labels, bins=10):
    e = 0.0; edges = np.linspace(0, 1, bins+1)
    for i in range(bins):
        m = (probs > edges[i]) & (probs <= edges[i+1])
        if m.sum():
            e += m.mean()*abs(probs[m].mean() - labels[m].mean())
    return e
z = np.random.randn(4000)*1.5
labels = (np.random.rand(4000) < sig(z)).astype(float)
logits = z*3.0
for T in [1, 2, 3, 4, 5]:
    print(T, round(ece(sig(logits/T), labels), 4))

105. What each one costs: Milestone 3 — temperature scaling

Trade off

106. The full program

Concept

import numpy as np
from sklearn.metrics import roc_auc_score, average_precision_score
# 1. ROC-AUC from scratch
scores = np.array([0.95,0.90,0.80,0.70,0.55,0.45,0.40,0.30,0.20,0.10])
y      = np.array([1,   1,   0,   1,   0,   1,   0,   0,   0,   0  ])
P, N = y.sum(), (1-y).sum(); tp = fp = 0; T = [0.0]; F = [0.0]
for i in np.argsort(-scores):
    tp += y[i]; fp += 1-y[i]; T.append(tp/P); F.append(fp/N)
auc = np.trapz(T, F)
print('ROC-AUC:', round(auc, 4), '== sklearn', bool(np.isclose(auc, roc_auc_score(y, scores))))
print('AP:', round(average_precision_score(y, scores), 4))
# 2. imbalance: ROC-AUC 0.736 rosy vs PR-AUC 0.111 honest
# 3. temperature scaling: ECE 0.1448 (T=1) -> 0.0176 (T=3)

If your ROC matches sklearn, PR-AUC exposes the imbalance, and temperature scaling minimizes ECE at T = 3 — you can evaluate a model honestly and trust its probabilities.

107. Fill in: value (verified) for The full program

Comparison

108. Show it off

Concept

Out loud, slides closed: explain (1) what one ROC point represents and why the curve steps up on positives and right on negatives, (2) why ROC-AUC flatters an imbalanced model while PR-AUC doesn't, and (3) what temperature scaling changes — and what it provably leaves alone.

Stretch: reproduce AUC = 0.875 by all three routes (trapezoid, sklearn, concordant-pair count), then plot a reliability diagram before and after temperature scaling. Calibration returns in LLM sampling temperature (Week 37) and uncertainty estimation.

109. Connect it up: Lesson 35: ROC, PR Curves & Calibration

Connect it up

110. What you can do now

Recap

• Read a confusion matrix and compute TPR = 0.75, FPR = 0.333, precision = 0.60 at a threshold
• Build the ROC curve (up on a positive, right on a negative) and get AUC = 0.875 three independent ways
• Interpret AUC as P(random positive > random negative) and flip a worse-than-random model (AUC → 1 − AUC)
• Use PR-AUC / average precision under imbalance — ROC-AUC 0.736 flatters, PR-AUC 0.111 tells the truth
• Define calibration and ECE, and drive ECE from 0.1448 to 0.0176 with temperature scaling at T = 3