Lesson 37: Phase 1 Review & Mock Prep

Discussion prompt

Before we open Lesson 37: Phase 1 Review & Mock Prep: without looking back, what was the main idea of Initialization & Gradient Clipping, and what could you do by the end of it that you could not do before?

Hint: One sentence for the idea, one for the skill. If the second one is blank, that is the part to revisit.

Comparison matrix

From The running dataset: refill the hours x column from what you know. The rest of the table is as it appeared.

Discussion prompt

Phase 1 taught linear algebra, probability, and optimization as separate weeks. The deep truth is that they are one idea wearing four costumes.

That is stated as though it always holds. Do one of two things: produce a case where it fails, or say precisely what rules such a case out. "It just does" is not on the menu.

Hint: Hunt at the extremes first — zero, one, negative, empty, equal. If every extreme survives, the reason they survive is the proof.

Put in order

Put the moves of Route 1 — the normal equations into the order they have to happen.

1. Start from the overdetermined system Xw = y
2. Left-multiply both sides by Xᵀ
3. Assemble the entries by hand
4. Verify: solve gives w = [2.2, 0.6]

Why: These are the moves of the worked example in the order it makes them, and each one is set up by the one before it. Five equations, two unknowns — no exact solution.

Annotate

From Route 1 — the normal equations — read this one piece at a time. What is each part doing?

On: \( X = \begin{bmatrix} 1 & 1 \\ 1 & 2 \\ 1 & 3 \\ 1 & 4 \\ 1 & 5 \end{bmatrix},\quad y = \begin{bmatrix} 2 \\ 4 \\ 5 \\ 4 \\ 5 \end{bmatrix} \)

• Five equations, two unknowns — no exact solution. Least squares finds the closest w.
• Xᵀ is (2×5), so XᵀX is a square (2×2) system — the normal equations from Lesson 7.
• XᵀX top-left = Σ1 = 5, off-diagonal = Σx = 15, bottom-right = Σx² = 55; Xᵀy = [Σy, Σxy] = [20, 66].

Discussion prompt

Route 1 — XᵀX and Xᵀy, entry by entry — before any calculation: what is the plan? Name the moves in order, in plain English, without doing the arithmetic.

Hint: It starts with: Top-left = ones·ones = Σ1 = 5

Discussion prompt

Work backwards. The example finished here:

Bottom-right = x·x = Σx² = 1+4+9+16+25 = 55; and Xᵀy = [Σy, Σxy] = [20, 66]

What was it asked to do, and what must it have been given? Reconstruct the problem from its answer.

Hint: Every quantity in the result had to enter somewhere. Account for each one.

Predict first

Self-contained: re-import NumPy, re-define x, y, X, then solve XᵀX w = Xᵀy. Never invert — solve factors the system.

Commit before you compute: what does Route 1 in code — solve the normal equations come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: solve returns [2.2, 0.6]

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. Line 5 builds XᵀX and Xᵀy and factors the system in one call.

Comparison matrix

From Route 1 in code — solve the normal equations: every row here is a choice with a cost. Fill the value (verified) column, then say which row you would actually pick and what you give up for it.

Discussion prompt

Explain Why squared error is a probability statement by analogy to something with no Machine Learning in it at all — a queue, a recipe, a map, a bank balance, whatever fits. Then say where your analogy breaks.

Hint: An analogy that never breaks is not an analogy, it is the same idea wearing a hat. Find the seam — that is the part that is actually new.

Put in order

Put the moves of Route 2 — OLS is Gaussian MLE into the order they have to happen.

1. Model the noise as Gaussian: y = Xw + ε, ε ~ N(0, σ²)
2. Write the negative log-likelihood
3. Minimize over w
4. Verify: the NLL is smallest exactly at w = [2.2, 0.6]

Why: These are the moves of the worked example in the order it makes them, and each one is set up by the one before it. Lesson 8: assume each residual is independent Gaussian noise around the linear mean.

Draw it

Draw what Route 2 — OLS is Gaussian MLE just did — the shape of it, not the line-by-line working. One picture, labels only where you need them. Then check it against the steps: anything you could not draw is a step you followed rather than understood.

Discussion prompt

Compute the OLS w, then show the Gaussian negative log-likelihood is minimized there — any nudge makes it larger.

What do you need to know — or decide — before the first line can be written? List everything the problem has to hand you.

Hint: Anything you would have to invent to get started is a thing the problem must supply.

Discussion prompt

Route 3 — the pseudoinverse (SVD) — before any calculation: what is the plan? Name the moves in order, in plain English, without doing the arithmetic.

Hint: It starts with: Recall the SVD: X = UΣVᵀ (Lesson 16)

Draw it

Translate both ways. First write the expression above as a sentence with no symbols in it at all. Then cover it, and write your sentence back as notation. If the two versions disagree, the disagreement is the thing to fix.

Predict first

The table runs: pseudoinverse X⁺y | L16 | w = [2.2, 0.6] · lstsq (QR/SVD) | L16 | w = [2.2, 0.6]

In Route 3 & 4 in code — pinv, lstsq, and the hat matrix, given the rows so far: what is the next one — the row where route is hat matrix Hy?

Correct: hat matrix Hy | L25 | ŷ = [2.8, 3.4, 4.0, 4.6, 5.2]

Why: The relationship between the columns, not the individual numbers, is what generates the next row. Hy is the projection of y onto the column space — the same predictions X@w produces.

Discussion prompt

Explain Why projection is 'closest' to a student a year behind you. No notation, no jargon they have not met — and it still has to be true.

Hint: If your explanation needs a symbol they have never seen, you are describing the notation rather than the idea.

Fill in the blanks

From Verify the projection properties — one line has had its right-hand side removed. Put it back.

import numpy as np
x = np.array([1., 2., 3., 4., 5.])
y = np.array([2., 4., 5., 4., 5.])
X = np.c_[np.ones(5), x]
H = X @ np.linalg.inv(X.T @ X) @ X.T
print(np.allclose(H, H @ H)) # True (idempotent)
print(np.allclose(H, H.T)) # True (symmetric)
print(round(np.trace(H), 4)) # 2.0
print((y - H @ y).round(4)) # residuals

Why: X is what everything below it consumes, so the wrong expression here fails later and somewhere else. All four checks pass. The residual (I−H)y sticks straight out of the column space — the geometric signature of the least-squares fit.

Sort into buckets

Sort these by value (verified), from memory, without looking back at Verify the projection properties. Telling them apart on the spot is the skill; the table is only where the answer happens to be written down.

True

H == H@H (idempotent); H == Hᵀ (symmetric)

2.0

trace(H)

[−0.8, 0.6, 1.0, −0.6, −0.2]

(I − H)y

g1

value (verified) is "True" for H == H@H (idempotent), H == Hᵀ (symmetric) — that is what the table on "Verify the projection properties" records, and it is the single property separating this group from the rest.

g2

value (verified) is "2.0" for trace(H) — that is what the table on "Verify the projection properties" records, and it is the single property separating this group from the rest.

g3

value (verified) is "[−0.8, 0.6, 1.0, −0.6, −0.2]" for (I − H)y — that is what the table on "Verify the projection properties" records, and it is the single property separating this group from the rest.

Step through it

Step through The fit in numbers — residuals and R² one row at a time. What is driving the change, and what would the row after the last one be?

1. Step 1: x is 1
2. Step 2: x is 2
3. Step 3: x is 3
4. Step 4: x is 4
5. Step 5: x is 5

Put in order

Put the moves of Eigenvalues by hand, then PD into the order they have to happen.

1. Write det(A − λI) = 0
2. Expand to λ² − 4λ + 3 = 0
3. Verify: both λ > 0, so A is positive definite

Why: These are the moves of the worked example in the order it makes them, and each one is set up by the one before it. Eigenvalues are the λ that make A − λI singular.

Annotate

From Eigenvalues by hand, then PD — read this one piece at a time. What is each part doing?

On: \( \lambda^2 - 4\lambda + 3 = (\lambda - 1)(\lambda - 3) = 0 \;\Rightarrow\; \lambda = 1,\, 3 \)

• Eigenvalues are the λ that make A − λI singular.
• (2−λ)² − 1 = 4 − 4λ + λ² − 1 = λ² − 4λ + 3. Note trace = 4 = sum of roots, det = 3 = product.
• np.roots([1,−4,3]) returns [3, 1] and eigh returns [1, 3] — matching the hand result. All-positive eigenvalues ⇒ PD (L13).

Fill in the blanks

KL ≥ 0 and cross-entropy = H + KL, with the scaffolding fading: two lines are gone now — fill both.

import numpy as np
p = np.array([0.7, 0.2, 0.1]) # true distribution
q = np.array([0.5, 0.3, 0.2]) # model distribution
H = -np.sum(p * np.log(p)) # entropy of p
KL = np.sum(p * np.log(p / q)) # KL(p || q) >= 0
CE = **-np.sum(p * np.log(q))** # cross-entropy H(p,q)
print(round(H, 4), round(KL, 4), round(CE, 4))
print(np.isclose(CE, H + KL)) # True

Why: Reproducing these unaided, rather than reading them, is what tells you the method has transferred. Minimizing cross-entropy over q minimizes KL (H(p) is fixed), so CE-loss drives the model toward the data distribution.

Step through it

Step through KL ≥ 0 and cross-entropy = H + KL one row at a time. What is driving the change, and what would the row after the last one be?

1. Step 1: quantity is H(p) entropy
2. Step 2: quantity is KL(p‖q)
3. Step 3: quantity is CE(p,q)
4. Step 4: quantity is H + KL

Predict first

The SSE gradient is ∇L = 2Xᵀ(Xw − y). Step downhill from zero; with a small enough learning rate it lands on the closed-form solution.

Commit before you compute: what does Gradient descent converges to the OLS w come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: After 20000 steps at lr = 0.001, GD w = [2.2, 0.6]

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. Convexity guarantees a single global minimum; GD slides into it and matches the closed form.

Comparison matrix

From Gradient descent converges to the OLS w: refill the w (verified) column from what you know. The rest of the table is as it appeared.

Predict first

A student writes this, and it looks reasonable:

Bigger steps should learn faster, so crank the learning rate to lr = 0.02 for the same SSE gradient descent.

It is wrong. Say what breaks — and say it before you turn the page.

Correct: The Hessian is 2XᵀX with largest eigenvalue L ≈ 118.

Pick lr < 2/L where L is the largest eigenvalue of the Hessian 2XᵀX.

Why: The Hessian is 2XᵀX with largest eigenvalue L ≈ 118. Stability needs lr < 2/L ≈ 0.017. At 0.02 each step overshoots and the iterate grows geometrically without bound — verified: after enough steps w overflows to inf.

Discussion prompt

The rule this trap just fixed:

Pick lr < 2/L where L is the largest eigenvalue of the Hessian 2XᵀX.

Now break it on purpose. Build a case that violates it and follow the consequences until something visibly fails. Where does the failure first show up — and would you have noticed it if you had not been looking?

Hint: The dangerous rules are the ones whose violation still produces an answer. If yours fails loudly, try to find one that fails quietly.

Fill in the blanks

From Newton solves OLS in ONE step — one line has had its right-hand side removed. Put it back.

import numpy as np
x = np.array([1., 2., 3., 4., 5.])
y = np.array([2., 4., 5., 4., 5.])
X = np.c_[np.ones(5), x]
w = np.array([0.0, 0.0]) # any start
grad = 2 * X.T @ (X @ w - y) # gradient at w
Hess = 2 * X.T @ X # Hessian (constant for OLS)
w_new = w - np.linalg.solve(Hess, grad) # one Newton step
print(w_new.round(4)) # [2.2 0.6]

Why: w is what everything below it consumes, so the wrong expression here fails later and somewhere else. For a quadratic the Hessian is constant and the second-order model is exact, so a single Newton step is the closed-form solution.

Sort into buckets

Sort these by value (verified), from memory, without looking back at softmax + cross-entropy ⇒ dL/dz = p − y. Telling them apart on the spot is the skill; the table is only where the answer happens to be written down.

[0.6590, 0.2424, 0.0986]

p = softmax(z)

[−0.3410, 0.2424, 0.0986]

p − y (analytic); numerical ∂L/∂z

True (atol 1e-5)

match

g1

value (verified) is "[0.6590, 0.2424, 0.0986]" for p = softmax(z) — that is what the table on "softmax + cross-entropy ⇒ dL/dz = p − y" records, and it is the single property separating this group from the rest.

g2

value (verified) is "[−0.3410, 0.2424, 0.0986]" for p − y (analytic), numerical ∂L/∂z — that is what the table on "softmax + cross-entropy ⇒ dL/dz = p − y" records, and it is the single property separating this group from the rest.

g3

value (verified) is "True (atol 1e-5)" for match — that is what the table on "softmax + cross-entropy ⇒ dL/dz = p − y" records, and it is the single property separating this group from the rest.

Fill in the blanks

ROC-AUC vs PR-AUC under 2% imbalance, with the scaffolding fading: two lines are gone now — fill both.

import numpy as np
from sklearn.metrics import roc_auc_score, average_precision_score
np.random.seed(0)
n = 5000
y = (np.random.rand(n) < 0.02).astype(int) # ~2% positives
scores = *np.random.rand(n) + 0.15y** # weak signal
print(int(y.sum()), 'of', n) # 105 of 5000
print(round(roc_auc_score(y, scores), 4)) # 0.6369
print(round(average_precision_score(y, scores), 4)) # 0.1771

Why: Reproducing these unaided, rather than reading them, is what tells you the method has transferred. ROC-AUC's false-positive rate has a huge true-negative denominator, so it stays flattering.

Predict first

A student writes this, and it looks reasonable:

Quick recall: a p-value of 0.03 means a 3% chance the null is true; det > 0 means positive definite; and MSE works fine for classification.

It is wrong. Say what breaks — and say it before you turn the page.

Correct: All three are the classic traps — the p-value reverses the conditioning, det>0 allows two negative eigenvalues (their product is positive), and MSE's gradient vanishes on confident wrong predictions.

You've beaten all three this phase — keep them straight.

Why: All three are the classic traps — the p-value reverses the conditioning, det>0 allows two negative eigenvalues (their product is positive), and MSE's gradient vanishes on confident wrong predictions.

Sort into buckets

Some of these hold up and some are the exact mistakes this lesson is built to prevent. Sort them.

Holds up

Design matrix X = [1, x] (a bias column then the feature), target y. Keep these five numbers in your head — every slide reuses them.; Phase 1 taught linear algebra, probability, and optimization as separate weeks. The deep truth is that they are one idea wearing four costumes.; Route 1 just chose to square the residuals. Route 2 explains why squaring is the right penalty — it falls out of a noise model, not a whim.

Breaks

Bigger steps should learn faster, so crank the learning rate to lr = 0.02 for the same SSE gradient descent.; Quick recall: a p-value of 0.03 means a 3% chance the null is true; det > 0 means positive definite; and MSE works fine for classification.

sound

These are stated as this lesson states them — each one survives the edge cases Lesson 37: Phase 1 Review & Mock Prep puts it through.

flawed

Each of these is lifted from a trap in this deck: reasonable-sounding, and wrong in a way that only shows up once you rely on it.

Predict first

The table runs: det(A) | 1.0 > 0 | misleadingly 'passes' · eigenvalues | [−1, −1] | both < 0 → NOT PD

In Trap in code — det > 0 is NOT positive definite, given the rows so far: what is the next one — the row where test is xᵀAx at [1,1]?

Correct: xᵀAx at [1,1] | −2.0 | confirms not PD

Why: The relationship between the columns, not the individual numbers, is what generates the next row. Two negatives multiply to a positive determinant, so det>0 is fooled.

Put in order

These are the steps of The exam-pace retrieval protocol, scrambled. Put them back in order before the next slide shows you.

1. Cold attempt first — slides closed, paper out (the struggle is the encoding)
2. Then check — reveal only after a genuine attempt
3. Reconstruct aloud — re-derive the result with the answer hidden again
4. Log every gap — any hint or miss goes on the retrieval list
5. Re-quiz on a schedule — 3 days, 1 week, 2 weeks (spaced retrieval)

Why: This is the order the recipe itself gives. Recalling the sequence without the slide in front of you is the difference between recognising the method and being able to run it — most of what goes wrong in practice is a step done out of turn.

Sort into buckets

These are the pieces of Lesson 37: Phase 1 Review & Mock Prep, out of order. Put each one back under the part of the lesson it belongs to.

One dataset, four routes to OLS

The running dataset; Why 'four ways' is the whole point; Route 1 — the normal equations

The four pillars, re-derived

Pillar 1 — Linear Algebra; Eigenvalues by hand, then PD; Eigen + PSD + dominant vector — the 60-second drill

Traps & synthesis

The dependency map; Trap in code — det > 0 is NOT positive definite; The exam-pace retrieval protocol

s1

One dataset, four routes to OLS is where Lesson 37: Phase 1 Review & Mock Prep puts The running dataset, Why 'four ways' is the whole point, Route 1 — the normal equations. Knowing which part of the lesson a problem belongs to is most of knowing which method to reach for.

s2

The four pillars, re-derived is where Lesson 37: Phase 1 Review & Mock Prep puts Pillar 1 — Linear Algebra, Eigenvalues by hand, then PD, Eigen + PSD + dominant vector — the 60-second drill. Knowing which part of the lesson a problem belongs to is most of knowing which method to reach for.

s3

Traps & synthesis is where Lesson 37: Phase 1 Review & Mock Prep puts The dependency map, Trap in code — det > 0 is NOT positive definite, The exam-pace retrieval protocol. Knowing which part of the lesson a problem belongs to is most of knowing which method to reach for.

Eliminate the wrong options

Why do the normal equations, Gaussian MLE, the pseudoinverse, and the hat-matrix projection all return the same w = [2.2, 0.6]?

3 of these 4 are wrong. Strike them one at a time, and say what rules each one out before you strike the next. The survivor is the answer.

• A. They are four descriptions of the same optimum: minimizing squared error = projecting y onto col(X) = maximizing the Gaussian likelihood
• B. It is a numerical coincidence specific to this 5-point dataset
• C. Only the normal equations are exact; the other three are approximations that happened to round to the same value
• D. They agree only because the data is perfectly linear

Survives elimination: A

Why: Least squares has one minimizer. The perpendicular-residual condition (projection), ∇SSE = 0 (normal equations), X⁺y (SVD), and the Gaussian MLE all characterize that same point, so they must coincide exactly when X has full column rank.

Check your understanding

Why do the normal equations, Gaussian MLE, the pseudoinverse, and the hat-matrix projection all return the same w = [2.2, 0.6]?

• A. They are four descriptions of the same optimum: minimizing squared error = projecting y onto col(X) = maximizing the Gaussian likelihood (correct)
• B. It is a numerical coincidence specific to this 5-point dataset
• C. Only the normal equations are exact; the other three are approximations that happened to round to the same value
• D. They agree only because the data is perfectly linear

Answer: A

Why: Least squares has one minimizer. The perpendicular-residual condition (projection), ∇SSE = 0 (normal equations), X⁺y (SVD), and the Gaussian MLE all characterize that same point, so they must coincide exactly when X has full column rank.

Why B tempts people

Not a coincidence — the equivalence is a theorem that holds for any full-column-rank X, not just these five points.

Why C tempts people

All four are exact (in exact arithmetic). pinv/lstsq differ only in floating-point stability, not in the answer they target.

Why D tempts people

The data is NOT perfectly linear here — the minimum error is 2.4, not 0. The routes agree precisely in the noisy, no-exact-fit case that least squares is built for.

Check your understanding

The singular values of any matrix A are:

• A. the non-negative square roots of the eigenvalues of AᵀA (correct)
• B. the eigenvalues of A
• C. the diagonal entries of A
• D. always equal to the eigenvalues in magnitude

Answer: A

Why: σᵢ = √λᵢ(AᵀA). AᵀA is PSD so the σ are real and ≥ 0, and the SVD exists for ANY matrix (L16). We verified [3.1926, 2.1926] = √eig(AᵀA) for a 3×2 A.

Why B tempts people

Non-square matrices have no eigenvalues yet always have singular values; even for square A the two generally differ.

Why C tempts people

Only true for an already-diagonal matrix; in general the diagonal is unrelated to the σ.

Why D tempts people

|λ| = σ only for normal matrices (e.g. symmetric); generally they differ.

Predict first

Minimizing mean squared error is maximum likelihood under which assumption?

Answer it in your own words, now, with nothing to choose from. The options are on the next slide — and picking the right one off a list is an easier skill than producing it.

Correct: Gaussian noise with constant variance

Why: With y = f(x) + ε, ε ~ N(0,σ²), the negative log-likelihood reduces to (1/2σ²)Σ(y−f)² — exactly MSE (L8). We saw the NLL bottom out at the OLS w = [2.2, 0.6].

Check your understanding

Minimizing mean squared error is maximum likelihood under which assumption?

• A. Gaussian noise with constant variance (correct)
• B. Laplace noise
• C. Bernoulli labels
• D. no assumption — MSE is always the likelihood

Answer: A

Why: With y = f(x) + ε, ε ~ N(0,σ²), the negative log-likelihood reduces to (1/2σ²)Σ(y−f)² — exactly MSE (L8). We saw the NLL bottom out at the OLS w = [2.2, 0.6].

Why B tempts people

Laplace noise gives the L1 / mean-absolute-error loss, not squared error.

Why C tempts people

Bernoulli labels give cross-entropy (classification), not MSE.

Why D tempts people

MSE corresponds to a specific (Gaussian) noise model; other models give other losses.

Predict first

You run gradient descent on the OLS loss with learning rate 0.02 and the weights explode to infinity. The largest eigenvalue of the Hessian 2XᵀX is about 118. What is the fix?

Answer it in your own words, now, with nothing to choose from. The options are on the next slide — and picking the right one off a list is an easier skill than producing it.

Correct: Lower the learning rate below 2/L ≈ 0.017

Why: Gradient descent on a quadratic is stable only when lr < 2/L, where L is the largest Hessian eigenvalue. Here 2/L ≈ 0.017, so lr = 0.02 overshoots every step and diverges; lr = 0.001 converges to [2.2, 0.6].

Check your understanding

You run gradient descent on the OLS loss with learning rate 0.02 and the weights explode to infinity. The largest eigenvalue of the Hessian 2XᵀX is about 118. What is the fix?

• A. Lower the learning rate below 2/L ≈ 0.017 (correct)
• B. Add more iterations — it will eventually converge
• C. Switch the loss to cross-entropy
• D. The loss is non-convex, so use a random restart

Answer: A

Why: Gradient descent on a quadratic is stable only when lr < 2/L, where L is the largest Hessian eigenvalue. Here 2/L ≈ 0.017, so lr = 0.02 overshoots every step and diverges; lr = 0.001 converges to [2.2, 0.6].

Why B tempts people

More steps of a divergent iteration diverge faster, not slower — the iterate grows geometrically each step.

Why C tempts people

Cross-entropy is for classification; it does not fit this regression and does not address the step-size instability.

Why D tempts people

The OLS loss is convex (Hessian 2XᵀX is PSD everywhere) — there are no bad local minima. The problem is purely the step size.

Eliminate the wrong options

On a 2%-positive dataset your model gets ROC-AUC 0.64 but PR-AUC 0.18. Which metric better reflects minority-class performance, and why?

3 of these 4 are wrong. Strike them one at a time, and say what rules each one out before you strike the next. The survivor is the answer.

• A. PR-AUC — its precision axis exposes false positives among the rare positives, which ROC-AUC's huge true-negative denominator hides
• B. ROC-AUC — it is always the industry standard
• C. Accuracy — it summarizes both in one number
• D. They are interchangeable; 0.64 and 0.18 describe the same thing

Survives elimination: A

Why: Under heavy imbalance ROC-AUC stays optimistic because the false-positive rate is diluted by many true negatives. PR-AUC watches precision on the positives and drops to 0.18, honestly reflecting the minority performance (verified on 105/5000 positives).

Check your understanding

On a 2%-positive dataset your model gets ROC-AUC 0.64 but PR-AUC 0.18. Which metric better reflects minority-class performance, and why?

• A. PR-AUC — its precision axis exposes false positives among the rare positives, which ROC-AUC's huge true-negative denominator hides (correct)
• B. ROC-AUC — it is always the industry standard
• C. Accuracy — it summarizes both in one number
• D. They are interchangeable; 0.64 and 0.18 describe the same thing

Answer: A

Why: Under heavy imbalance ROC-AUC stays optimistic because the false-positive rate is diluted by many true negatives. PR-AUC watches precision on the positives and drops to 0.18, honestly reflecting the minority performance (verified on 105/5000 positives).

Why B tempts people

ROC-AUC being common does not make it appropriate here — it is precisely the metric that misleads under imbalance.

Why C tempts people

Accuracy scores 98% by predicting all-negative and finding zero positives — it hides the failure entirely.

Why D tempts people

They diverge sharply here (0.64 vs 0.18); that gap is the whole point — they measure different things under imbalance.

Fill in the blanks

From Milestone 3 — score R² and match sklearn — one line has had its right-hand side removed. Put it back.

import numpy as np
from sklearn.linear_model import LinearRegression
x = np.array([1., 2., 3., 4., 5.])
y = np.array([2., 4., 5., 4., 5.])
X = np.c_[np.ones_like(x), x]
w = np.linalg.solve(X.T @ X, X.T @ y)
yhat = X @ w
r2 = 1 - np.sum((y - yhat)2) / np.sum((y - y.mean())2)
print('w =', w.round(4), ' R2 =', round(r2, 4)) # [2.2 0.6] 0.6
m = LinearRegression().fit(x.reshape(-1, 1), y)
print('sklearn:', round(m.intercept_, 4), round(m.coef_[0], 4))

Why: X is what everything below it consumes, so the wrong expression here fails later and somewhere else. Your from-scratch line explains 60% of the score variance, and sklearn's fitted intercept and slope match your hand-built coefficients exactly.

Discussion prompt

Work backwards. The example finished here:

R² = 0.6, and sklearn prints 2.2 0.6 — everything agrees

What was it asked to do, and what must it have been given? Reconstruct the problem from its answer.

Hint: Every quantity in the result had to enter somewhere. Account for each one.

Discussion prompt

Explain Ridge — the always-invertible fallback to a student a year behind you. No notation, no jargon they have not met — and it still has to be true.

Hint: If your explanation needs a symbol they have never seen, you are describing the notation rather than the idea.

Discussion prompt

Same collinear Xc that makes solve raise. Show the singular eigenvalue, add λI, and watch it become solvable — runnable as written.

What do you need to know — or decide — before the first line can be written? List everything the problem has to hand you.

Hint: Anything you would have to invent to get started is a thing the problem must supply.

Comparison matrix

From Ridge lifts the zero eigenvalue: every row here is a choice with a cost. Fill the eigenvalues (verified) column, then say which row you would actually pick and what you give up for it.

Predict first

One more Phase-1 bridge: adding λ‖w‖² is the same as putting a Gaussian prior w ~ N(0, τ²I) on the weights and doing MAP (L12). The penalty is the log-prior. Ridge shrinks ‖w‖; verify it.

Commit before you compute: what does Ridge is MAP with a Gaussian prior come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: λ = 1 shrinks w from [2.2, 0.6] to [1.1712, 0.8649], and ‖w‖ from 2.2804 to 1.4559

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. The Gaussian prior pulls weights toward 0, trading a little training fit for stability — the MAP estimate under a zero-mean weight prior.

Comparison matrix

From Ridge is MAP with a Gaussian prior: refill the w = [w₀, w₁] (verified) column from what you know. The rest of the table is as it appeared.

Discussion prompt

Explain Mock-exam readiness by analogy to something with no Machine Learning in it at all — a queue, a recipe, a map, a bank balance, whatever fits. Then say where your analogy breaks.

Hint: An analogy that never breaks is not an analogy, it is the same idea wearing a hat. Find the seam — that is the part that is actually new.

Discussion prompt

Slides closed, out loud: explain (1) why the four OLS routes must agree, (2) why softmax+CE gives p − y, and (3) what λI does to the eigenvalues of XᵀX.

That is stated as though it always holds. Do one of two things: produce a case where it fails, or say precisely what rules such a case out. "It just does" is not on the menu.

Hint: Hunt at the extremes first — zero, one, negative, empty, equal. If every extreme survives, the reason they survive is the proof.

Draw it

One page, no notation unless you need it: draw how these connect — One dataset, four routes to OLS · The four pillars, re-derived · Traps & synthesis · Retrieval drills · Find the gaps & build it. Put an arrow wherever one of them is what makes another possible, and label the arrow with why.