Lesson 26: Evaluation Metrics

Discussion prompt

Before we open Lesson 26: Evaluation Metrics: without looking back, what was the main idea of Projections & the Hat Matrix, and what could you do by the end of it that you could not do before?

Hint: One sentence for the idea, one for the skill. If the second one is blank, that is the part to revisit.

Comparison matrix

From The running example: a spam classifier: refill the true y column from what you know. The rest of the table is as it appeared.

Predict first

Apply pred = (p ≥ 0.5). Two of the five spam emails clear 0.5 easily (0.95, 0.90, 0.80, 0.60) but e5 at 0.30 does not; two ham emails (e6 0.70, e7 0.55) wrongly clear it. This is our prediction vector.

Commit before you compute: what does Threshold the scores at t = 0.5 come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: pred = [1 1 1 1 0 1 1 0 0 0 0 0]

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. Four of five spam correctly flagged (e1–e4), e5 missed; two ham wrongly flagged (e6, e7).

Comparison matrix

From Threshold the scores at t = 0.5: every row here is a choice with a cost. Fill the p ≥ 0.5 → pred column, then say which row you would actually pick and what you give up for it.

Discussion prompt

Every (prediction, truth) pair falls into exactly one of four boxes. Positive/negative names what the model said; true/false names whether it was right.

That is stated as though it always holds. Do one of two things: produce a case where it fails, or say precisely what rules such a case out. "It just does" is not on the menu.

Hint: Hunt at the extremes first — zero, one, negative, empty, equal. If every extreme survives, the reason they survive is the proof.

Discussion prompt

A boolean mask like (pred==1)&(y==1) is True exactly on the true positives; .sum() counts the Trues. Build all four counts from pred and y:

What do you need to know — or decide — before the first line can be written? List everything the problem has to hand you.

Hint: Anything you would have to invent to get started is a thing the problem must supply.

Discussion prompt

Work backwards. The example finished here:

TP=4, FP=2, FN=1, TN=5

What was it asked to do, and what must it have been given? Reconstruct the problem from its answer.

Hint: Every quantity in the result had to enter somewhere. Account for each one.

Discussion prompt

Explain Reading the names: true/false, positive/negative by analogy to something with no Machine Learning in it at all — a queue, a recipe, a map, a bank balance, whatever fits. Then say where your analogy breaks.

Hint: An analogy that never breaks is not an analogy, it is the same idea wearing a hat. Find the seam — that is the part that is actually new.

Discussion prompt

Explain Precision and recall: two different questions to a student a year behind you. No notation, no jargon they have not met — and it still has to be true.

Hint: If your explanation needs a symbol they have never seen, you are describing the notation rather than the idea.

Predict first

The table runs: precision | TP/(TP+FP) | 4/6 | 0.6667 · recall | TP/(TP+FN) | 4/5 | 0.8000

In Precision and recall on our counts, given the rows so far: what is the next one — the row where metric is accuracy?

Correct: accuracy | (TP+TN)/n | 9/12 | 0.7500

Why: The relationship between the columns, not the individual numbers, is what generates the next row. Of the 6 emails we flagged as spam, 4 truly were — the 2 false alarms (e6, e7) drag precision down to two-thirds.

Step through it

Step through Precision and recall on our counts one row at a time. What is driving the change, and what would the row after the last one be?

1. Step 1: metric is precision
2. Step 2: metric is recall
3. Step 3: metric is accuracy

Put in order

Put the moves of F1 by hand, then verified in code into the order they have to happen.

1. Numerator: 2PR = 2·(4/6)·(4/5) = 2·(16/30) = 32/30
2. Denominator: P + R = 4/6 + 4/5 = 20/30 + 24/30 = 44/30
3. F1 = (32/30)/(44/30) = 32/44 = 0.7273

Why: These are the moves of the worked example in the order it makes them, and each one is set up by the one before it. Multiply the two rates, double it.

Annotate

Walk the callouts on F1 by hand, then verified in code. Each one is a place this is easy to get subtly wrong.

• Multiply the two rates, double it. Keep fractions to stay exact.
• Common denominator 30. The 30s cancel in the ratio.
• F1 sits between recall (0.8) and precision (0.667), pulled toward the smaller — the signature of a harmonic mean.

Put in order

These are the steps of From confusion matrix to metrics, scrambled. Put them back in order before the next slide shows you.

1. Threshold: turn scores into labels with pred = (p ≥ t)
2. Count: four boolean masks → TP, FP, FN, TN (they must sum to n)
3. Precision = TP/(TP+FP) — punishes false alarms
4. Recall = TP/(TP+FN) — punishes misses
5. F1 = 2PR/(P+R) — the harmonic mean, high only when both are

Why: This is the order the recipe itself gives. Recalling the sequence without the slide in front of you is the difference between recognising the method and being able to run it — most of what goes wrong in practice is a step done out of turn.

Predict first

A student writes this, and it looks reasonable:

A fraud detector reports 99% accuracy on a stream that is 99% legitimate. Ship it — it's almost never wrong.

It is wrong. Say what breaks — and say it before you turn the page.

Correct: Predicting 'negative' for EVERYONE scores 99% accuracy while catching zero fraud.

On imbalanced data report recall, F1, or AUC-PR — metrics that stare at the minority class instead of being drowned out by the majority.

Why: Predicting 'negative' for EVERYONE scores 99% accuracy while catching zero fraud. Accuracy rewards a model that has learned nothing about the rare class you actually care about.

Fill in the blanks

From Macro and micro F1 diverge on a rare class — one line has had its right-hand side removed. Put it back.

import numpy as np
from sklearn.metrics import f1_score
y = np.array([0,0,0,0,1,1,2]) # class 2 is rare (1 example)
pred = np.array([0,0,0,1,1,1,0])
print(round(f1_score(y,pred,average="macro"),4),
round(f1_score(y,pred,average="micro"),4))

Why: pred is what everything below it consumes, so the wrong expression here fails later and somewhere else. Class 2's F1 is 0, and macro averages it in with equal weight, dragging the score down.

Fill in the blanks

Sweep the threshold: watch P and R trade, with the scaffolding fading: two lines are gone now — fill both.

import numpy as np
y = np.array([1,1,1,1,1,0,0,0,0,0,0,0])
p = np.array([0.95,0.90,0.80,0.60,0.30,0.70,0.55,0.40,0.20,0.15,0.10,0.05])
for t in [0.25, 0.50, 0.75]:
pred = (p >= t).astype(int)
TP = ((pred==1)&(y==1)).sum(); FP = ((pred==1)&(y==0)).sum()
FN = ((pred==0)&(y==1)).sum()
prec = TP/(TP+FP); rec = TP/(TP+FN)
print(t, int(TP), int(FP), int(FN), round(prec,3), round(rec,3))

Why: Reproducing these unaided, rather than reading them, is what tells you the method has transferred. Lowering the bar admits more true spam (recall up) but also more ham false alarms (precision down).

Step through it

Step through Sweep the threshold: watch P and R trade one row at a time. What is driving the change, and what would the row after the last one be?

1. Step 1: t is 0.75
2. Step 2: t is 0.50
3. Step 3: t is 0.25

Sort into buckets

These are the pieces of Lesson 26: Evaluation Metrics, out of order. Put each one back under the part of the lesson it belongs to.

The confusion matrix

The running example: a spam classifier; Threshold the scores at t = 0.5; Four outcomes, four names

Precision, recall, F1

Precision and recall: two different questions; The knob you can always cheat; Precision and recall on our counts

The threshold trade-off

0.5 is a choice, not a law; Sweep the threshold: watch P and R trade; The ROC curve: TPR vs FPR

s1

The confusion matrix is where Lesson 26: Evaluation Metrics puts The running example: a spam classifier, Threshold the scores at t = 0.5, Four outcomes, four names. Knowing which part of the lesson a problem belongs to is most of knowing which method to reach for.

s2

Precision, recall, F1 is where Lesson 26: Evaluation Metrics puts Precision and recall: two different questions, The knob you can always cheat, Precision and recall on our counts. Knowing which part of the lesson a problem belongs to is most of knowing which method to reach for.

s3

The threshold trade-off is where Lesson 26: Evaluation Metrics puts 0.5 is a choice, not a law, Sweep the threshold: watch P and R trade, The ROC curve: TPR vs FPR. Knowing which part of the lesson a problem belongs to is most of knowing which method to reach for.

Step through it

Step through Trace the ROC curve points and watch the numbers move. Now imagine the input ten times bigger: which column is the one that stops this being practical?

1. Step 1: threshold t is 1.00
2. Step 2: threshold t is 0.75
3. Step 3: threshold t is 0.65
4. Step 4: threshold t is 0.50
5. Step 5: threshold t is 0.35
6. Step 6: threshold t is 0.00

Discussion prompt

AUC by hand: count the losing pairs — before any calculation: what is the plan? Name the moves in order, in plain English, without doing the arithmetic.

Hint: It starts with: 0.60 loses to 0.70 → 1 losing pair

Draw it

Draw what AUC by hand: count the losing pairs just did — the shape of it, not the line-by-line working. One picture, labels only where you need them. Then check it against the steps: anything you could not draw is a step you followed rather than understood.

Predict first

A student writes this, and it looks reasonable:

"AUC = 0.8857, so the model is about 89% accurate."

It is wrong. Say what breaks — and say it before you turn the page.

Correct: AUC never looks at a threshold, so it says nothing about how many LABELS are right.

Read AUC as a ranking probability: P(random spam scores above random ham) = 0.8857.

Why: AUC never looks at a threshold, so it says nothing about how many LABELS are right. Our model's accuracy at t=0.5 is 0.75, not 0.8857 — different numbers measuring different things.

Discussion prompt

The rule this trap just fixed:

AUC is threshold-free (0.8857 here); accuracy depends entirely on the cut (0.75 at t=0.5, different at every other t). A model can have high AUC yet poor accuracy if you threshold it badly.

Now break it on purpose. Build a case that violates it and follow the consequences until something visibly fails. Where does the failure first show up — and would you have noticed it if you had not been looking?

Hint: The dangerous rules are the ones whose violation still produces an answer. If yours fails loudly, try to find one that fails quietly.

Fill in the blanks

From AUC-PR: when negatives swamp positives — one line has had its right-hand side removed. Put it back.

import numpy as np
from sklearn.metrics import average_precision_score, roc_auc_score
rng = np.random.default_rng(0)
y = *np.array([1,1,1] + [0]27)** # 3 positives, 27 negatives (10%)
p = np.concatenate([[0.80,0.60,0.40], rng.uniform(0,0.7,27)])
print(round(roc_auc_score(y,p),4), round(average_precision_score(y,p),4))

Why: y is what everything below it consumes, so the wrong expression here fails later and somewhere else. ROC-AUC looks decent; AUC-PR reveals the model is barely better than the 0.1 base rate at surfacing the rare positives.

Predict first

Two models, both right on every label on four examples, but one is confident (0.9/0.1) and one is timid (0.6/0.4). Log-loss separates them; accuracy would call them identical.

Commit before you compute: what does Log-loss rewards calibration come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: Timid: every term is −ln(0.6) = 0.5108 → log-loss 0.5108

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. Same correct labels, but only 0.6 confidence, so each term is −ln(0.6).

Predict first

A student writes this, and it looks reasonable:

"Being 0.99 sure costs the same whether I'm right or wrong — it's just one prediction."

It is wrong. Say what breaks — and say it before you turn the page.

Correct: Log-loss on that one wrong-but-certain email is −ln(1−0.99) = −ln(0.01) ≈ 4.6 — one confidently-wrong prediction can outweigh dozens of correct ones.

Calibrate: only claim 0.99 when you would be right 99 times in 100. Log-loss forces this discipline — it is unbounded above as p → 0 on a true positive.

Why: Log-loss on that one wrong-but-certain email is −ln(1−0.99) = −ln(0.01) ≈ 4.6 — one confidently-wrong prediction can outweigh dozens of correct ones.

Comparison matrix

From Trap: a confident wrong answer is cheap: refill the log-loss column from what you know. The rest of the table is as it appeared.

Step through it

Step through The running regression example: house prices one row at a time. What is driving the change, and what would the row after the last one be?

1. Step 1: house is h1
2. Step 2: house is h2
3. Step 3: house is h3
4. Step 4: house is h4
5. Step 5: house is h5
6. Step 6: house is h6

Put in order

Put the moves of MSE and RMSE, entry by entry into the order they have to happen.

1. Squared errors: 0.25, 1.00, 0.00, 1.00, 1.00, 1.00 → sum 4.25
2. MSE = 4.25 / 6 = 0.7083
3. RMSE = √0.7083 = 0.8416

Why: These are the moves of the worked example in the order it makes them, and each one is set up by the one before it. (−0.5)²=0.25, (+1)²=1, 0²=0, (−1)²=1, (+1)²=1, (−1)²=1.

Draw it

Draw what MSE and RMSE, entry by entry just did — the shape of it, not the line-by-line working. One picture, labels only where you need them. Then check it against the steps: anything you could not draw is a step you followed rather than understood.

Put in order

Put the moves of MAE and R², entry by entry into the order they have to happen.

1. MAE = (0.5+1+0+1+1+1)/6 = 4.5/6 = 0.75
2. ȳ = 28/6 = 4.6667; SS_tot = Σ(yt − ȳ)² = 23.333
3. R² = 1 − SS_res/SS_tot = 1 − 4.25/23.333 = 0.8179

Why: These are the moves of the worked example in the order it makes them, and each one is set up by the one before it. Absolute values, then average. Every miss counts in proportion to its size — no extra weight on the big ones.

Annotate

Walk the callouts on MAE and R², entry by entry. Each one is a place this is easy to get subtly wrong.

• Absolute values, then average. Every miss counts in proportion to its size — no extra weight on the big ones.
• Deviations of yt=[2,3,5,4,6,8] from 4.6667 squared and summed give 23.333 — the error of the dumb mean-only baseline.
• The model removes about 82% of the baseline's squared error — it explains 82% of the variance in price.

Discussion prompt

Explain Why RMSE ≥ MAE, always to a student a year behind you. No notation, no jargon they have not met — and it still has to be true.

Hint: If your explanation needs a symbol they have never seen, you are describing the notation rather than the idea.

Predict first

A student writes this, and it looks reasonable:

"RMSE and MAE both average error, so report whichever — they tell the same story."

It is wrong. Say what breaks — and say it before you turn the page.

Correct: One big miss barely moves MAE but explodes RMSE, because RMSE squares it.

Choose by how you want large errors treated: RMSE to punish big misses, MAE to stay robust to outliers.

Why: One big miss barely moves MAE but explodes RMSE, because RMSE squares it. They diverge exactly when outliers exist — which is precisely when the choice of metric matters most.

Sort into buckets

Some of these hold up and some are the exact mistakes this lesson is built to prevent. Sort them.

Holds up

A score is not yet a decision. To turn p into a predicted label we pick a threshold — flag as spam when p ≥ t. We start with the default t = 0.5.; Every (prediction, truth) pair falls into exactly one of four boxes. Positive/negative names what the model said; true/false names whether it was right.; Precision's enemy is the false alarm (FP); recall's enemy is the miss (FN). Same TP on top — they differ only in which error type sits in the denominator.

Breaks

A fraud detector reports 99% accuracy on a stream that is 99% legitimate. Ship it — it's almost never wrong.; "AUC = 0.8857, so the model is about 89% accurate."

sound

These are stated as this lesson states them — each one survives the edge cases Lesson 26: Evaluation Metrics puts it through.

flawed

Each of these is lifted from a trap in this deck: reasonable-sounding, and wrong in a way that only shows up once you rely on it.

Fill in the blanks

MAPE and its blow-up near zero, with the scaffolding fading: two lines are gone now — fill both.

import numpy as np
yt = np.array([100.0, 50.0, 0.5]) # third true value is tiny
yp = np.array([110.0, 45.0, 1.0])
per = *np.abs((yt-yp)/yt)100**
print([float(round(v,1)) for v in per], round(float(per.mean()),2))

Why: Reproducing these unaided, rather than reading them, is what tells you the method has transferred. The first two houses are off by a sane 10%.

Sort into buckets

Sort these by |e|/yt · 100, from memory, without looking back at MAPE and its blow-up near zero. Telling them apart on the spot is the skill; the table is only where the answer happens to be written down.

10.0%

100.0; 50.0

100.0% ← blow-up

0.5

40.0%

MAPE (mean)

g1

|e|/yt · 100 is "10.0%" for 100.0, 50.0 — that is what the table on "MAPE and its blow-up near zero" records, and it is the single property separating this group from the rest.

g2

|e|/yt · 100 is "100.0% ← blow-up" for 0.5 — that is what the table on "MAPE and its blow-up near zero" records, and it is the single property separating this group from the rest.

g3

|e|/yt · 100 is "40.0%" for MAPE (mean) — that is what the table on "MAPE and its blow-up near zero" records, and it is the single property separating this group from the rest.

Discussion prompt

Run Metric-selection cheat sheet with this step confiscated:

Calibrated probabilities: log-loss (punishes confident-wrong)

Is it still possible? If it is, say what takes its place and what it costs you. If it is not, say exactly what that step was providing that nothing else does.

Hint: A step you can drop for free was never load-bearing. If you cannot drop it, name the thing that goes wrong the moment it is gone.

Discussion prompt

Metric-selection cheat sheet works on the cases you have just seen. Push it to the edge: what is the most degenerate input it still handles — empty, zero, one item, everything equal — and what is the first case where it stops being true? Name the case, not just "it breaks".

Hint: Try the smallest legal input, then the largest, then the one where two things collide. Methods are specified at their edges; the middle takes care of itself.

Fill in the blanks

From Perplexity: how surprised is the model? — one line has had its right-hand side removed. Put it back.

import numpy as np
# model's probability on each true next token, 5-token sentence
q = np.array([0.5, 0.25, 0.5, 0.1, 0.4])
nll = -np.mean(np.log(q)) # average negative log-likelihood
ppl = np.exp(nll)
print(round(nll,4), round(ppl,4))

Why: ppl is what everything below it consumes, so the wrong expression here fails later and somewhere else. The model is, on average, about as uncertain as a fair choice among ~3.3 tokens.

Draw it

Translate both ways. First write the expression above as a sentence with no symbols in it at all. Then cover it, and write your sentence back as notation. If the two versions disagree, the disagreement is the thing to fix.

Predict first

The table runs: the | 3 | 2 | 2 · cat | 1 | 1 | 1 · mat | 1 | 1 | 1

In BLEU idea: clipped n-gram precision, given the rows so far: what is the next one — the row where word is total / p1?

Correct: total / p1 | 5 | — | 4 → 0.8

Why: The relationship between the columns, not the individual numbers, is what generates the next row. Without clipping, the three 'the's would each count and precision would be inflated.

Sort into buckets

Sort these by cand count, from memory, without looking back at BLEU idea: clipped n-gram precision. Telling them apart on the spot is the skill; the table is only where the answer happens to be written down.

3

the

1

cat; mat

5

total / p1

g1

cand count is "3" for the — that is what the table on "BLEU idea: clipped n-gram precision" records, and it is the single property separating this group from the rest.

g2

cand count is "1" for cat, mat — that is what the table on "BLEU idea: clipped n-gram precision" records, and it is the single property separating this group from the rest.

g3

cand count is "5" for total / p1 — that is what the table on "BLEU idea: clipped n-gram precision" records, and it is the single property separating this group from the rest.

Discussion prompt

Where BLEU asks 'how much of my output is in the reference?' (precision), ROUGE asks 'how much of the reference did I cover?' (recall) — the natural fit for summarization, where you want to capture the key content.

What do you need to know — or decide — before the first line can be written? List everything the problem has to hand you.

Hint: Anything you would have to invent to get started is a thing the problem must supply.

Predict first

For image generation there is no reference per image. FID embeds real and generated images into a feature space and measures the distance between the two Gaussians (means μ, covariances Σ). Lower = generated set looks more like real. A 1-D toy shows the formula:

Commit before you compute: what does FID idea: distance between distributions come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: FID = 0.25 + (1 + 1.44 − 2·1.2) = 0.25 + 0.04 = 0.29

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. The mean gap contributes (0.5)²=0.25; the variance mismatch contributes 1+1.44−2√(1.44)=0.04.

Discussion prompt

Work backwards. The example finished here:

FID = 0.25 + (1 + 1.44 − 2·1.2) = 0.25 + 0.04 = 0.29

What was it asked to do, and what must it have been given? Reconstruct the problem from its answer.

Hint: Every quantity in the result had to enter somewhere. Account for each one.

Eliminate the wrong options

A model has precision = 0.8 and recall = 0.6. Its F1 score is:

3 of these 4 are wrong. Strike them one at a time, and say what rules each one out before you strike the next. The survivor is the answer.

• A. ≈ 0.686
• B. 0.70 (the average)
• C. 0.48 (the product)
• D. 1.40 (the sum)

Survives elimination: A

Why: F1 = 2PR/(P+R) = 2(0.8)(0.6)/(0.8+0.6) = 0.96/1.4 ≈ 0.686. The harmonic mean sits below the arithmetic mean, pulled toward the smaller value (recall).

Check your understanding

A model has precision = 0.8 and recall = 0.6. Its F1 score is:

• A. ≈ 0.686 (correct)
• B. 0.70 (the average)
• C. 0.48 (the product)
• D. 1.40 (the sum)

Answer: A

Why: F1 = 2PR/(P+R) = 2(0.8)(0.6)/(0.8+0.6) = 0.96/1.4 ≈ 0.686. The harmonic mean sits below the arithmetic mean, pulled toward the smaller value (recall).

Why B tempts people

0.70 is the ARITHMETIC mean (0.8+0.6)/2. F1 is the HARMONIC mean, always ≤ the arithmetic mean, and it penalizes imbalance between P and R.

Why C tempts people

0.48 = P·R is just the product. F1 doubles the product and divides by the sum P+R.

Why D tempts people

1.40 = P+R is the denominator of the F1 formula, not the score itself — you stopped one step early.

Predict first

A dataset is 99% negative. A model predicts 'negative' for every example. Which is true?

Answer it in your own words, now, with nothing to choose from. The options are on the next slide — and picking the right one off a list is an easier skill than producing it.

Correct: Accuracy is 0.99 but recall for the positive class is 0

Why: Predicting all-negative is right on every true negative (99% of the data) so accuracy = 0.99, but it catches zero positives, so recall = TP/(TP+FN) = 0 and F1 = 0. Accuracy hides a useless model.

Check your understanding

A dataset is 99% negative. A model predicts 'negative' for every example. Which is true?

• A. Accuracy is 0.99 but recall for the positive class is 0 (correct)
• B. Both accuracy and recall are 0.99
• C. Accuracy is 0.50
• D. F1 is 0.99

Answer: A

Why: Predicting all-negative is right on every true negative (99% of the data) so accuracy = 0.99, but it catches zero positives, so recall = TP/(TP+FN) = 0 and F1 = 0. Accuracy hides a useless model.

Why B tempts people

Recall = TP/(TP+FN) = 0/(all positives) = 0, not 0.99 — no positive is ever caught, so recall cannot be high.

Why C tempts people

Accuracy is the fraction correct = 990/1000 = 0.99, not 0.50. The model is right on every negative.

Why D tempts people

F1 = 2PR/(P+R) needs nonzero recall; with recall 0 (and precision undefined/0), F1 = 0, not 0.99.

Predict first

A classifier has AUC-ROC = 0.90. The most precise interpretation is:

Commit to an answer, then rate it — certain, fairly sure, or guessing — and write the rating down before you turn the page.

Correct: A random positive is scored higher than a random negative 90% of the time

Why: AUC-ROC equals P(random positive scored above random negative) — a threshold-free measure of ranking quality. Equivalently it is the area under the ROC curve.

The rating matters as much as the answer: confident-and-wrong is the combination that survives revision, because nothing about it feels like it needs revisiting.

Check your understanding

A classifier has AUC-ROC = 0.90. The most precise interpretation is:

• A. A random positive is scored higher than a random negative 90% of the time (correct)
• B. The model is 90% accurate
• C. 90% of predicted probabilities exceed 0.9
• D. Precision equals 0.90 at the 0.5 threshold

Answer: A

Why: AUC-ROC equals P(random positive scored above random negative) — a threshold-free measure of ranking quality. Equivalently it is the area under the ROC curve.

Why B tempts people

Accuracy depends on a chosen threshold; AUC is threshold-free and measures ranking, not the fraction of correct labels. Our running model has AUC 0.8857 but accuracy 0.75.

Why C tempts people

AUC ignores the absolute probability values entirely — only their relative ORDERING between the two classes matters.

Why D tempts people

Precision is a single-threshold quantity; AUC aggregates ranking across ALL thresholds and never fixes one.

Predict first

You compute RMSE = 1.23 and MAE = 0.63 on the same predictions. What does the large gap most likely indicate?

Answer it in your own words, now, with nothing to choose from. The options are on the next slide — and picking the right one off a list is an easier skill than producing it.

Correct: A few large errors (outliers) are inflating RMSE relative to MAE

Why: RMSE squares errors, so a few big misses pull it well above MAE. A wide RMSE/MAE ratio (here ~1.95) is a signature of outliers or high error variance.

Check your understanding

You compute RMSE = 1.23 and MAE = 0.63 on the same predictions. What does the large gap most likely indicate?

• A. A few large errors (outliers) are inflating RMSE relative to MAE (correct)
• B. A bug — RMSE can never exceed MAE
• C. The errors are all identical in size
• D. R² must be negative

Answer: A

Why: RMSE squares errors, so a few big misses pull it well above MAE. A wide RMSE/MAE ratio (here ~1.95) is a signature of outliers or high error variance.

Why B tempts people

RMSE ≥ MAE always (Jensen's inequality); a gap is expected, not a bug. RMSE below MAE would be the impossible case.

Why C tempts people

Identical error sizes give RMSE = MAE exactly (ratio 1). A large gap means the opposite — errors vary a lot.

Why D tempts people

R² measures variance explained and is unrelated to the RMSE/MAE gap; a model can have a large gap and still a high positive R².

Eliminate the wrong options

Model A outputs 0.9 on the correct class every time; Model B outputs 0.6 on the correct class every time. Both are 100% accurate. Which has the lower (better) log-loss?

3 of these 4 are wrong. Strike them one at a time, and say what rules each one out before you strike the next. The survivor is the answer.

• A. Model A — it is more confident on the correct class, so −ln(p) is smaller
• B. Model B — hedging is safer
• C. They tie, because accuracy is identical
• D. Neither — log-loss ignores probabilities

Survives elimination: A

Why: Log-loss per example is −ln(p) on the correct class: −ln(0.9)=0.105 for A vs −ln(0.6)=0.511 for B. A is ~5× lower, so A is better. Log-loss rewards calibrated confidence, which accuracy cannot see.

Check your understanding

Model A outputs 0.9 on the correct class every time; Model B outputs 0.6 on the correct class every time. Both are 100% accurate. Which has the lower (better) log-loss?

• A. Model A — it is more confident on the correct class, so −ln(p) is smaller (correct)
• B. Model B — hedging is safer
• C. They tie, because accuracy is identical
• D. Neither — log-loss ignores probabilities

Answer: A

Why: Log-loss per example is −ln(p) on the correct class: −ln(0.9)=0.105 for A vs −ln(0.6)=0.511 for B. A is ~5× lower, so A is better. Log-loss rewards calibrated confidence, which accuracy cannot see.

Why B tempts people

Hedging raises log-loss when you are right: −ln(0.6) > −ln(0.9). Confidence on the correct class is rewarded, not punished, as long as you are right.

Why C tempts people

Accuracy is blind to probabilities; log-loss is exactly the metric that separates two equally-accurate models by their confidence.

Why D tempts people

The opposite — log-loss is computed entirely from the predicted probabilities via −[y ln p + (1−y) ln(1−p)].

Discussion prompt

Explain Project: a metrics library from scratch by analogy to something with no Machine Learning in it at all — a queue, a recipe, a map, a bank balance, whatever fits. Then say where your analogy breaks.

Hint: An analogy that never breaks is not an analogy, it is the same idea wearing a hat. Find the seam — that is the part that is actually new.

Step through it

Step through Milestone 1 — the four counts one row at a time. What is driving the change, and what would the row after the last one be?

1. Step 1: quantity is TP, FP, FN, TN
2. Step 2: quantity is precision
3. Step 3: quantity is recall

Comparison matrix

From Milestone 3 — AUC vs sklearn: every row here is a choice with a cost. Fill the AUC column, then say which row you would actually pick and what you give up for it.

Comparison matrix

From The full program: refill the value column from what you know. The rest of the table is as it appeared.

Discussion prompt

Slides closed, out loud: explain (1) why accuracy fails on 99%-negative data, (2) what AUC-ROC means as a probability and how the 31/35 pairs produced 0.8857, and (3) when RMSE diverges from MAE.

That is stated as though it always holds. Do one of two things: produce a case where it fails, or say precisely what rules such a case out. "It just does" is not on the menu.

Hint: Hunt at the extremes first — zero, one, negative, empty, equal. If every extreme survives, the reason they survive is the proof.

Draw it

One page, no notation unless you need it: draw how these connect — The confusion matrix · Precision, recall, F1 · The threshold trade-off · Regression metrics · Generation metrics · Check yourself. Put an arrow wherever one of them is what makes another possible, and label the arrow with why.