Lesson 25: Projections & the Hat Matrix

Discussion prompt

Before we open Lesson 25: Projections & the Hat Matrix: without looking back, what was the main idea of Constrained Optimization & KKT, and what could you do by the end of it that you could not do before?

Hint: One sentence for the idea, one for the skill. If the second one is blank, that is the part to revisit.

Discussion prompt

Least squares found the weights w★ that make Xw★ the closest point in the column space of X to the target y. The residual came out perpendicular to that space.

That is stated as though it always holds. Do one of two things: produce a case where it fails, or say precisely what rules such a case out. "It just does" is not on the menu.

Hint: Hunt at the extremes first — zero, one, negative, empty, equal. If every extreme survives, the reason they survive is the proof.

Comparison matrix

From The 5-student running example: refill the hours x column from what you know. The rest of the table is as it appeared.

Discussion prompt

Explain H is fixed before you ever see y by analogy to something with no Machine Learning in it at all — a queue, a recipe, a map, a bank balance, whatever fits. Then say where your analogy breaks.

Hint: An analogy that never breaks is not an analogy, it is the same idea wearing a hat. Find the seam — that is the part that is actually new.

Discussion prompt

Explain A shadow on the floor to a student a year behind you. No notation, no jargon they have not met — and it still has to be true.

Hint: If your explanation needs a symbol they have never seen, you are describing the notation rather than the idea.

Put in order

Put the moves of Eigenvalues of a projector are only 0 and 1 into the order they have to happen.

1. Apply P to both sides of Pv = λv
2. But P² = P, so P²v = Pv = λv
3. Since v ≠ 0, λ² − λ = 0, so λ ∈ {0, 1}

Why: These are the moves of the worked example in the order it makes them, and each one is set up by the one before it. P(Pv) = P(λv) = λ(Pv) = λ·λv = λ²v.

Annotate

From Eigenvalues of a projector are only 0 and 1 — read this one piece at a time. What is each part doing?

On: \( P^2 v = \lambda^2 v \)

• P(Pv) = P(λv) = λ(Pv) = λ·λv = λ²v. The left side is P²v.
• Idempotence lets us replace P²v with λv on the left.
• λ(λ − 1) = 0. A projector's eigenvalues can ONLY be 0 (directions it kills) or 1 (directions it keeps). Nothing in between.

Predict first

Before the hat matrix, meet a projector you can see. P = [[1,0],[0,0]] flattens any 2-D point onto the x-axis: it keeps the first coordinate, zeroes the second.

Commit before you compute: what does The simplest projector: drop onto the x-axis come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: P²=P, Pᵀ=P, eig = [0, 1], trace = 1 = rank

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. Every projector law shows up in this 2×2 toy: idempotent, symmetric, eigenvalues 0/1, and trace 1 counting the one kept dimension (the x-axis).

Comparison matrix

From The simplest projector: drop onto the x-axis: every row here is a choice with a cost. Fill the value (verified) column, then say which row you would actually pick and what you give up for it.

Predict first

A student writes this, and it looks reasonable:

A projector 'undoes' cleanly, so surely P² = I — apply it twice and you're back where you started.

It is wrong. Say what breaks — and say it before you turn the page.

Correct: P² = I describes an INVOLUTION — a reflection (eigenvalues ±1), which is invertible (it's its own inverse).

A projector flattens — the second application has nothing left to do, so P² = P, not I.

Why: P² = I describes an INVOLUTION — a reflection (eigenvalues ±1), which is invertible (it's its own inverse). That is a totally different animal from a projection.

Discussion prompt

Derive H from the normal equations — before any calculation: what is the plan? Name the moves in order, in plain English, without doing the arithmetic.

Hint: It starts with: Start from the fitted values

Discussion prompt

Work backwards. The example finished here:

Group everything that touches y

What was it asked to do, and what must it have been given? Reconstruct the problem from its answer.

Hint: Every quantity in the result had to enter somewhere. Account for each one.

Sort into buckets

These are the pieces of Lesson 25: Projections & the Hat Matrix, out of order. Put each one back under the part of the lesson it belongs to.

The idea: least squares as a matrix

Where we left off in Lesson 7; The 5-student running example; H is fixed before you ever see y

Projection matrices

Idempotent: projecting twice = once; Symmetric: the projection is orthogonal; Eigenvalues of a projector are only 0 and 1

Building the hat matrix

The question the hat matrix answers; Derive H from the normal equations; Watch the shapes

s1

The idea: least squares as a matrix is where Lesson 25: Projections & the Hat Matrix puts Where we left off in Lesson 7, The 5-student running example, H is fixed before you ever see y. Knowing which part of the lesson a problem belongs to is most of knowing which method to reach for.

s2

Projection matrices is where Lesson 25: Projections & the Hat Matrix puts Idempotent: projecting twice = once, Symmetric: the projection is orthogonal, Eigenvalues of a projector are only 0 and 1. Knowing which part of the lesson a problem belongs to is most of knowing which method to reach for.

s3

Building the hat matrix is where Lesson 25: Projections & the Hat Matrix puts The question the hat matrix answers, Derive H from the normal equations, Watch the shapes. Knowing which part of the lesson a problem belongs to is most of knowing which method to reach for.

Put in order

Put the moves of One entry of H, by hand into the order they have to happen.

1. Recall (XᵀX)⁻¹ = [[1.1, −0.3], [−0.3, 0.1]]
2. Row 1 of X is [1, 1]; multiply it by (XᵀX)⁻¹
3. Dot that with row 1 again → H₁₁ = 0.6

Why: These are the moves of the worked example in the order it makes them, and each one is set up by the one before it. From Lesson 7: det(XᵀX) = 50, and the 2×2 inverse is (1/50)[[55,−15],[−15,5]] = [[1.1,−0.3],[−0.3,0.1]].

Annotate

Walk the callouts on One entry of H, by hand. Each one is a place this is easy to get subtly wrong.

• From Lesson 7: det(XᵀX) = 50, and the 2×2 inverse is (1/50)[[55,−15],[−15,5]] = [[1.1,−0.3],[−0.3,0.1]].
• Student 1 has x=1, so its row is [1, 1]. [1,1]·[[1.1,−0.3],[−0.3,0.1]] = [1.1−0.3, −0.3+0.1] = [0.8, −0.2].
• [0.8, −0.2]·[1, 1]ᵀ = 0.8 − 0.2 = 0.6. That is exactly the top-left of H and the leverage of student 1.

Discussion prompt

Assemble H for our 5-student data and print it. Note the code uses inv only to expose the structure — in production you'd use solve; more on that in the trap.

What do you need to know — or decide — before the first line can be written? List everything the problem has to hand you.

Hint: Anything you would have to invent to get started is a thing the problem must supply.

Step through it

Step through Build H in code — and see it one row at a time. One of these columns never changes — find it, and say why it cannot.

1. Step 1: is row1
2. Step 2: is row2
3. Step 3: is row3
4. Step 4: is row4
5. Step 5: is row5

Predict first

The table runs: 1 | 2 | 2.8 | 2.8 · 2 | 4 | 3.4 | 3.4 · 3 | 5 | 4.0 | 4.0 · 4 | 4 | 4.6 | 4.6

In H really produces the OLS fit, given the rows so far: what is the next one — the row where x is 5?

Correct: 5 | 5 | 5.2 | 5.2

Why: The relationship between the columns, not the individual numbers, is what generates the next row. The hat matrix reproduces the regression line's predictions to machine precision.

Step through it

Step through H really produces the OLS fit one row at a time. What is driving the change, and what would the row after the last one be?

1. Step 1: x is 1
2. Step 2: x is 2
3. Step 3: x is 3
4. Step 4: x is 4
5. Step 5: x is 5

Put in order

Put the moves of Prove H² = H algebraically into the order they have to happen.

1. Write H·H with the full formula
2. The inner XᵀX meets its own inverse
3. What's left is H again

Why: These are the moves of the worked example in the order it makes them, and each one is set up by the one before it. Substitute H = X(XᵀX)⁻¹Xᵀ into both factors.

Draw it

Draw what Prove H² = H algebraically just did — the shape of it, not the line-by-line working. One picture, labels only where you need them. Then check it against the steps: anything you could not draw is a step you followed rather than understood.

Fill in the blanks

From Prove Hᵀ = H algebraically — one line has had its right-hand side removed. Put it back.

import numpy as np
x = np.array([1., 2., 3., 4., 5.])
X = np.c_[np.ones(5), x]
H = X @ np.linalg.inv(X.T @ X) @ X.T
print("H.T == H:", np.allclose(H, H.T))

Why: X is what everything below it consumes, so the wrong expression here fails later and somewhere else. (ABC)ᵀ = CᵀBᵀAᵀ. Here A = X, B = (XᵀX)⁻¹, C = Xᵀ.

Draw it

Translate both ways. First write the expression above as a sentence with no symbols in it at all. Then cover it, and write your sentence back as notation. If the two versions disagree, the disagreement is the thing to fix.

Fill in the blanks

Eigenvalues and rank of H, numerically, with the scaffolding fading: two lines are gone now — fill both.

import numpy as np
x = np.array([1., 2., 3., 4., 5.])
X = np.c_[np.ones(5), x]
H = X @ np.linalg.inv(X.T @ X) @ X.T
print("eigenvalues:", np.linalg.eigvalsh(H).round(6))
print("rank(H) =", np.linalg.matrix_rank(H))
print("trace(H) =", round(np.trace(H), 4))

Why: Reproducing these unaided, rather than reading them, is what tells you the method has transferred. Exactly two nonzero (unit) eigenvalues — one per parameter (intercept + slope).

Predict first

A student writes this, and it looks reasonable:

H is 5×5 and packed with nonzero entries, so it's an invertible matrix like any other — just call inv(H).

It is wrong. Say what breaks — and say it before you turn the page.

Correct: H has eigenvalues 0 (three of them).

H is a projector, so it is singular by design — its rank is only p, not n.

Why: H has eigenvalues 0 (three of them). Any matrix with a zero eigenvalue has det = 0 and no inverse. inv(H) raises 'Singular matrix'.

Discussion prompt

The rule this trap just fixed:

H maps the whole 5-D data space into the 2-D column space, so 3 directions die. There is nothing to invert — that information is gone. Its eigenvalues are 0 and 1, never all nonzero.

Now break it on purpose. Build a case that violates it and follow the consequences until something visibly fails. Where does the failure first show up — and would you have noticed it if you had not been looking?

Hint: The dangerous rules are the ones whose violation still produces an answer. If yours fails loudly, try to find one that fails quietly.

Discussion prompt

Read the picture before the words. What is this showing, and what is the one thing it is built to make obvious? Commit to an answer, then read on.

Hint: Name the parts, then say what changes between them — and if nothing changes, say what is being held still.

Predict first

Prove (I − H)² = I − H using only H² = H. Expand the square like ordinary binomials (matrices commute with I).

Commit before you compute: what does I − H is a projector too come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: Replace H² with H

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. Idempotence of H: H² = H. So −2H + H² = −2H + H = −H.

Annotate

From I − H is a projector too — read this one piece at a time. What is each part doing?

On: \( (I - H)^2 = I - 2H + H = I - H \)

• (I − H)(I − H) = I·I − I·H − H·I + H·H = I − H − H + H².
• Idempotence of H: H² = H. So −2H + H² = −2H + H = −H.

Fill in the blanks

From Residuals ⊥ fitted values — one line has had its right-hand side removed. Put it back.

import numpy as np
x = np.array([1., 2., 3., 4., 5.])
y = np.array([2., 4., 5., 4., 5.])
X = np.c_[np.ones(5), x]
H = X @ np.linalg.inv(X.T @ X) @ X.T
M = np.eye(5) - H # residual projector
r = M @ y
yhat = H @ y
print("r =", r.round(4))
print("r . yhat =", round(float(r @ yhat), 10))
print("trace(I-H) =", round(np.trace(M), 4)) # n - p = 5 - 2

Why: H is what everything below it consumes, so the wrong expression here fails later and somewhere else. The residuals are exactly Lesson 7's, and they have zero overlap with the fit — H·(I−H) = 0 because HᵀM = H(I−H) = H − H² = 0.

Draw it

Draw what Residuals ⊥ fitted values just did — the shape of it, not the line-by-line working. One picture, labels only where you need them. Then check it against the steps: anything you could not draw is a step you followed rather than understood.

Fill in the blanks

The Pythagorean sum-of-squares split, with the scaffolding fading: two lines are gone now — fill both.

import numpy as np
x = np.array([1., 2., 3., 4., 5.])
y = np.array([2., 4., 5., 4., 5.])
X = np.c_[np.ones(5), x]
H = X @ np.linalg.inv(X.T @ X) @ X.T
yhat = H @ y
r = y - yhat
print("||y||^2 =", round(float(y @ y), 4))
print("||yhat||^2 =", round(float(yhat @ yhat), 4))
print("||r||^2 (RSS) =", round(float(r @ r), 4))
print("split holds:", np.allclose(y @ y, yhat @ yhat + r @ r))

Why: Reproducing these unaided, rather than reading them, is what tells you the method has transferred. The total energy of y splits cleanly into the fitted part (83.6) and the residual part (2.4 = the minimized error).

Comparison matrix

From The Pythagorean sum-of-squares split: refill the value (verified) column from what you know. The rest of the table is as it appeared.

Sort into buckets

Every line below is part of the definition of orthogonal projection matrix or of leverage — one or the other, never both. Put each where it belongs.

orthogonal projection matrix

A matrix P with BOTH P² = P and Pᵀ = P.; Idempotence makes it a projection; symmetry makes the leftover (v − Pv) perpendicular to the target subspace.

leverage

It measures how strongly point i's own yᵢ pulls its own fitted ŷᵢ.; High leverage = an extreme x-position with lots of pull on the line.

b1

A matrix P with BOTH P² = P and Pᵀ = P. Idempotence makes it a projection; symmetry makes the leftover (v − Pv) perpendicular to the target subspace. Least squares uses exactly this kind.

b2

hᵢᵢ = the diagonal of H. It measures how strongly point i's own yᵢ pulls its own fitted ŷᵢ. High leverage = an extreme x-position with lots of pull on the line. It depends on X only — never on y.

Fill in the blanks

From Read leverage off diag(H) — one line has had its right-hand side removed. Put it back.

import numpy as np
x = np.array([1., 2., 3., 4., 5.])
X = np.c_[np.ones(5), x]
H = X @ np.linalg.inv(X.T @ X) @ X.T
lev = np.diag(H) # leverage = diagonal of H
print("leverage =", lev.round(4))
print("sum =", round(float(lev.sum()), 4), " trace =", round(np.trace(H), 4))

Why: H is what everything below it consumes, so the wrong expression here fails later and somewhere else. The two endpoints tie at 0.6 (most pull), the center point is lowest at 0.2, and they sum to the number of parameters.

Predict first

The table runs: 1 | 0.2 | 0.4 | 0.6 | 0.6 · 3 | 0.2 | 0.0 | 0.2 | 0.2

In Leverage the closed-form way, given the rows so far: what is the next one — the row where x is 5?

Correct: 5 | 0.2 | 0.4 | 0.6 | 0.6

Why: The relationship between the columns, not the individual numbers, is what generates the next row. x=1: 0.2 + 4/10 = 0.6. x=3: 0.2 + 0/10 = 0.2.

Discussion prompt

Work backwards. The example finished here:

The closed form equals diag(H) exactly

What was it asked to do, and what must it have been given? Reconstruct the problem from its answer.

Hint: Every quantity in the result had to enter somewhere. Account for each one.

Discussion prompt

RSS = 2.4, n−p = 3, so s² = 0.8. Feed residuals and leverage into the formula. Which point is most influential?

What do you need to know — or decide — before the first line can be written? List everything the problem has to hand you.

Hint: Anything you would have to invent to get started is a thing the problem must supply.

Step through it

Step through Compute Cook's distance for all five points one row at a time. What is driving the change, and what would the row after the last one be?

1. Step 1: point is 1
2. Step 2: point is 2
3. Step 3: point is 3
4. Step 4: point is 4
5. Step 5: point is 5

Predict first

Cook's distance predicts point 1 shifts the fit most. Test it directly — refit with each point deleted and compare w to the full [2.2, 0.6].

Commit before you compute: what does Prove it: leave one point out come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: Drop point 3 → w = [1.95, 0.6]: barely moves

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. The center point had the biggest residual but tiny leverage, so deleting it hardly touches the line — matching its small Cook's D of 0.20.

Annotate

Walk the callouts on Prove it: leave one point out. Each one is a place this is easy to get subtly wrong.

• Removing the high-influence endpoint moves the intercept from 2.2 to 3.8 and the slope from 0.6 to 0.2. That massive shift is exactly the D = 1.5 warning made real.
• The center point had the biggest residual but tiny leverage, so deleting it hardly touches the line — matching its small Cook's D of 0.20. Influence, not residual, is what matters.

Discussion prompt

Explain Reading Cook's distance to a student a year behind you. No notation, no jargon they have not met — and it still has to be true.

Hint: If your explanation needs a symbol they have never seen, you are describing the notation rather than the idea.

Predict first

A student writes this, and it looks reasonable:

Point 1 and point 5 both have leverage 0.6, the highest — they're outliers dragging the fit, so drop them.

It is wrong. Say what breaks — and say it before you turn the page.

Correct: Leverage is computed from X alone — it's pure x-position, not error.

Judge points by Cook's distance, which needs leverage and a residual. Only point 1 is actually influential.

Why: Leverage is computed from X alone — it's pure x-position, not error. Point 5 sits nearly on the line (residual −0.2, Cook's D 0.09); deleting it just throws away good data and widens your intervals.

Sort into buckets

Some of these hold up and some are the exact mistakes this lesson is built to prevent. Sort them.

Holds up

Least squares found the weights w★ that make Xw★ the closest point in the column space of X to the target y. The residual came out perpendicular to that space.; We reuse Lesson 7's data so every number connects: five students, hours x and score y, fit by ŷ = w₀ + w₁·x.; The hat matrix is built from X alone — the feature positions. It does not depend on the targets y at all.

Breaks

A projector 'undoes' cleanly, so surely P² = I — apply it twice and you're back where you started.; H is 5×5 and packed with nonzero entries, so it's an invertible matrix like any other — just call inv(H).

sound

These are stated as this lesson states them — each one survives the edge cases Lesson 25: Projections & the Hat Matrix puts it through.

flawed

Each of these is lifted from a trap in this deck: reasonable-sounding, and wrong in a way that only shows up once you rely on it.

Put in order

These are the steps of The projection / hat-matrix toolkit, scrambled. Put them back in order before the next slide shows you.

1. Projector: P² = P (idempotent) and Pᵀ = P (symmetric) ⇒ eigenvalues 0/1, singular, rank = tr(P)
2. Hat matrix: H = X(XᵀX)⁻¹Xᵀ; the fit is ŷ = Hy — no need to solve for w
3. Residuals: r = (I − H)y, orthogonal to the fit (r·ŷ = 0); tr(I − H) = n − p
4. Leverage: hᵢᵢ = diag(H), sums to tr(H) = p; closed form 1/n + (xᵢ−x̄)²/Sxx
5. Influence: Cook's Dᵢ combines leverage and residual — flag high D, don't just delete high h

Why: This is the order the recipe itself gives. Recalling the sequence without the slide in front of you is the difference between recognising the method and being able to run it — most of what goes wrong in practice is a step done out of turn.

Discussion prompt

The projection / hat-matrix toolkit works on the cases you have just seen. Push it to the edge: what is the most degenerate input it still handles — empty, zero, one item, everything equal — and what is the first case where it stops being true? Name the case, not just "it breaks".

Hint: Try the smallest legal input, then the largest, then the one where two things collide. Methods are specified at their edges; the middle takes care of itself.

Eliminate the wrong options

An orthogonal projection matrix P satisfies which pair of properties?

3 of these 4 are wrong. Strike them one at a time, and say what rules each one out before you strike the next. The survivor is the answer.

• A. P² = P and Pᵀ = P
• B. P² = I and Pᵀ = P
• C. P is invertible and Pᵀ = P⁻¹
• D. det(P) = 1 and P² = P

Survives elimination: A

Why: Idempotent (P² = P: projecting twice equals once) plus symmetric (Pᵀ = P: the drop is perpendicular). Together these force the eigenvalues to be only 0 and 1, so P is singular whenever it isn't the identity.

Check your understanding

An orthogonal projection matrix P satisfies which pair of properties?

• A. P² = P and Pᵀ = P (correct)
• B. P² = I and Pᵀ = P
• C. P is invertible and Pᵀ = P⁻¹
• D. det(P) = 1 and P² = P

Answer: A

Why: Idempotent (P² = P: projecting twice equals once) plus symmetric (Pᵀ = P: the drop is perpendicular). Together these force the eigenvalues to be only 0 and 1, so P is singular whenever it isn't the identity.

Why B tempts people

P² = I is an involution (a reflection, eigenvalues ±1), which is invertible — the opposite of a projector. A projector obeys P² = P, not P² = I.

Why C tempts people

A non-trivial projector is singular (it has zero eigenvalues), so it has no inverse at all — P⁻¹ doesn't exist. Orthogonal here refers to the perpendicular drop, not to P being an orthogonal (rotation) matrix.

Why D tempts people

det(P) = 0 for any projector onto a proper subspace, because it has at least one zero eigenvalue — never det = 1. Idempotence alone (P² = P) is right, but the determinant claim is wrong.

Predict first

For the hat matrix H of a regression with n data points and p parameters (intercept included), tr(H) equals:

Answer it in your own words, now, with nothing to choose from. The options are on the next slide — and picking the right one off a list is an easier skill than producing it.

Correct: p — the number of parameters / dimension of the column space

Why: H projects onto the p-dimensional column space, so its eigenvalues are p ones and (n−p) zeros. The trace is the sum of eigenvalues, hence p — the model's degrees of freedom. For our data that is 2.

Check your understanding

For the hat matrix H of a regression with n data points and p parameters (intercept included), tr(H) equals:

• A. p — the number of parameters / dimension of the column space (correct)
• B. n — the number of data points
• C. n − p — the residual degrees of freedom
• D. 1, since H is idempotent

Answer: A

Why: H projects onto the p-dimensional column space, so its eigenvalues are p ones and (n−p) zeros. The trace is the sum of eigenvalues, hence p — the model's degrees of freedom. For our data that is 2.

Why B tempts people

n is tr(I), the trace of the full identity. H only keeps the p column-space directions; the other n−p go to zero, so its trace is p, not n.

Why C tempts people

n − p is tr(I − H), the trace of the RESIDUAL projector — the residual degrees of freedom. tr(H) is its complement, p.

Why D tempts people

Idempotence forces eigenvalues into {0, 1} but says nothing about how many 1s there are. A rank-p projector has p ones, so tr(H) = p; it equals 1 only for a single-parameter model.

Predict first

Two points share the same high leverage hᵢᵢ = 0.6, but point 1 has residual −0.8 and point 5 has residual −0.2. What follows?

Answer it in your own words, now, with nothing to choose from. The options are on the next slide — and picking the right one off a list is an easier skill than producing it.

Correct: Point 1 is far more influential — high leverage AND a large residual give it a big Cook's distance

Why: Cook's distance multiplies the squared residual by a leverage factor, so it's large only when both are large. Point 1 (h=0.6, r=−0.8) scores D=1.50; point 5 (h=0.6, r=−0.2) scores just 0.09 — and deleting point 1 swings w from [2.2,0.6] to [3.8,0.2].

Check your understanding

Two points share the same high leverage hᵢᵢ = 0.6, but point 1 has residual −0.8 and point 5 has residual −0.2. What follows?

• A. Point 1 is far more influential — high leverage AND a large residual give it a big Cook's distance (correct)
• B. They are equally influential, since leverage alone determines influence
• C. Point 5 is more influential because it sits closer to the line
• D. Neither is influential, because high leverage means the point is trustworthy

Answer: A

Why: Cook's distance multiplies the squared residual by a leverage factor, so it's large only when both are large. Point 1 (h=0.6, r=−0.8) scores D=1.50; point 5 (h=0.6, r=−0.2) scores just 0.09 — and deleting point 1 swings w from [2.2,0.6] to [3.8,0.2].

Why B tempts people

Leverage is only the lever arm (x-position); influence also needs a residual. Equal leverage with different residuals gives very different Cook's distances — 1.50 vs 0.09 here.

Why C tempts people

Sitting near the line means a SMALL residual, which makes a point LESS influential, not more. Point 5's small residual is exactly why its Cook's distance is tiny.

Why D tempts people

High leverage is not a certificate of trustworthiness — it's potential to move the line. Whether that potential is realized depends on the residual, which is why point 1 is dangerous and point 5 is not.

Eliminate the wrong options

You call np.linalg.inv(H) on a 5×5 hat matrix from a 2-parameter fit. What happens?

3 of these 4 are wrong. Strike them one at a time, and say what rules each one out before you strike the next. The survivor is the answer.

• A. LinAlgError — H is singular (three zero eigenvalues, det = 0)
• B. It returns H⁻¹, which equals H since H is idempotent
• C. It returns the identity, since H projects onto everything
• D. It returns (XᵀX)⁻¹, undoing the projection

Survives elimination: A

Why: H has eigenvalues [0,0,0,1,1]; three zeros mean det(H) = 0 and no inverse exists. inv(H) raises 'Singular matrix'. A projector (other than the identity) is always singular — it destroys the n−p residual directions, and that information can't be recovered.

Check your understanding

You call np.linalg.inv(H) on a 5×5 hat matrix from a 2-parameter fit. What happens?

• A. LinAlgError — H is singular (three zero eigenvalues, det = 0) (correct)
• B. It returns H⁻¹, which equals H since H is idempotent
• C. It returns the identity, since H projects onto everything
• D. It returns (XᵀX)⁻¹, undoing the projection

Answer: A

Why: H has eigenvalues [0,0,0,1,1]; three zeros mean det(H) = 0 and no inverse exists. inv(H) raises 'Singular matrix'. A projector (other than the identity) is always singular — it destroys the n−p residual directions, and that information can't be recovered.

Why B tempts people

Idempotence (H² = H) is not invertibility. If H⁻¹ existed, H² = H would give H = I — but H ≠ I here (it has rank 2, not 5), so no inverse exists.

Why C tempts people

H projects onto the 2-D column space, not onto all of ℝ⁵. Only I projects onto everything; H sends three directions to zero, which is why it can't be inverted.

Why D tempts people

(XᵀX)⁻¹ is a real, invertible 2×2 matrix, but it is not H⁻¹ — H is 5×5 and singular. The presence of (XᵀX)⁻¹ inside H's formula does not make H itself invertible.

Discussion prompt

Explain Project: hat matrix, residuals & leverage by analogy to something with no Machine Learning in it at all — a queue, a recipe, a map, a bank balance, whatever fits. Then say where your analogy breaks.

Hint: An analogy that never breaks is not an analogy, it is the same idea wearing a hat. Find the seam — that is the part that is actually new.

Step through it

Step through Milestone 1 — build & verify H one row at a time. What is driving the change, and what would the row after the last one be?

1. Step 1: check is H@H==H, H.T==H
2. Step 2: check is rank(H)
3. Step 3: check is trace(H)

Comparison matrix

From Milestone 3 — leverage: every row here is a choice with a cost. Fill the leverage column, then say which row you would actually pick and what you give up for it.

Comparison matrix

From The full program: refill the value column from what you know. The rest of the table is as it appeared.

Discussion prompt

Slides closed, out loud: explain (1) why H² = H (the (XᵀX)⁻¹(XᵀX) cancellation), (2) why residuals are perpendicular to fitted values, and (3) what tr(H) counts geometrically.

That is stated as though it always holds. Do one of two things: produce a case where it fails, or say precisely what rules such a case out. "It just does" is not on the menu.

Hint: Hunt at the extremes first — zero, one, negative, empty, equal. If every extreme survives, the reason they survive is the proof.

Draw it

One page, no notation unless you need it: draw how these connect — The idea: least squares as a matrix · Projection matrices · Building the hat matrix · Residuals & the I − H projector · Leverage & the trace · Influence: Cook's distance. Put an arrow wherever one of them is what makes another possible, and label the arrow with why.