Lesson 34: Concentration Inequalities & Generalization

Discussion prompt

Before we open Lesson 34: Concentration Inequalities & Generalization: without looking back, what was the main idea of Gradient Checking & Debugging, and what could you do by the end of it that you could not do before?

Hint: One sentence for the idea, one for the skill. If the second one is blank, that is the part to revisit.

Discussion prompt

One example threads the whole lesson. We flip a coin whose true heads-probability is p = 0.3 (we call heads 1, tails 0). We never see p; we only see flips and form the sample mean X̄.

That is stated as though it always holds. Do one of two things: produce a case where it fails, or say precisely what rules such a case out. "It just does" is not on the menu.

Hint: Hunt at the extremes first — zero, one, negative, empty, equal. If every extreme survives, the reason they survive is the proof.

Predict first

Flip the p = 0.3 coin 20 times with a fixed seed and read off X̄. Watch it miss 0.3. Runnable as written:

Commit before you compute: what does One sample already strays come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: 8 heads in 20 flips → X̄ = 0.40, not 0.30

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. A finite sample is noisy: the estimate landed 0.10 above the truth.

Comparison matrix

From One sample already strays: refill the value (verified) column from what you know. The rest of the table is as it appeared.

Discussion prompt

Explain Why this is the whole game by analogy to something with no Machine Learning in it at all — a queue, a recipe, a map, a bank balance, whatever fits. Then say where your analogy breaks.

Hint: An analogy that never breaks is not an analogy, it is the same idea wearing a hat. Find the seam — that is the part that is actually new.

Discussion prompt

Explain The one inequality behind Markov to a student a year behind you. No notation, no jargon they have not met — and it still has to be true.

Hint: If your explanation needs a symbol they have never seen, you are describing the notation rather than the idea.

Put in order

Put the moves of Derive Markov, step by step into the order they have to happen.

1. Start from the pointwise inequality
2. Take expectations of both sides
3. Pull the constant a out; E of an indicator is a probability
4. Divide both sides by a > 0

Why: These are the moves of the worked example in the order it makes them, and each one is set up by the one before it. Holds for every outcome, as we checked case by case.

Annotate

From Derive Markov, step by step — read this one piece at a time. What is each part doing?

On: \( a \cdot \mathbf{1}[X \ge a] \le X \)

• Holds for every outcome, as we checked case by case.
• Expectation is monotone: if U ≤ V pointwise then E[U] ≤ E[V]. The inequality survives.
• E[1[A]] = P(A) by definition, so E[a·1[X≥a]] = a·P(X≥a).

Discussion prompt

Take X ~ Exponential(1), so E[X] = 1. Draw two million samples and compare the empirical right-tail P(X ≥ a) to the Markov ceiling 1/a. Complete snippet:

What do you need to know — or decide — before the first line can be written? List everything the problem has to hand you.

Hint: Anything you would have to invent to get started is a thing the problem must supply.

Comparison matrix

From Markov holds — and how loosely: every row here is a choice with a cost. Fill the empirical P(X ≥ a) column, then say which row you would actually pick and what you give up for it.

Predict first

A student writes this, and it looks reasonable:

Markov gives P(X ≥ 4) ≤ 0.25, so roughly a quarter of the mass sits above 4.

It is wrong. Say what breaks — and say it before you turn the page.

Correct: An upper bound is a CEILING, not an estimate.

Read 0.25 as a worst-case ceiling. To get near the true 0.018, feed the bound more structure.

Why: An upper bound is a CEILING, not an estimate. The true value is 0.0183 — about 14× smaller. Markov used only the mean, so it is deliberately pessimistic.

Fill in the blanks

From Derive Chebyshev from Markov — finish the line. Write what belongs on the right of the equals sign before you look.

P(|X-\mu| \ge k\sigma) = P\big((X-\mu)^2 \ge (k\sigma)^2\big)

Why: Producing the right-hand side unprompted is the difference between recognising this line and being able to use it. |X − μ| ≥ kσ happens for exactly the same outcomes as (X − μ)² ≥ (kσ)² — squaring both non-negative sides preserves the event.

Draw it

Translate both ways. First write the expression above as a sentence with no symbols in it at all. Then cover it, and write your sentence back as notation. If the two versions disagree, the disagreement is the thing to fix.

Predict first

The table runs: P(|X − 1| ≥ 2) | 0.0498 · P(Y ≥ 4), Y = (X−1)² | 0.0498 · E[Y] (= Var = σ²) | 1.0

In Verify Chebyshev really is Markov-on-Y, given the rows so far: what is the next one — the row where quantity is Markov E[Y]/4 = 1/k²?

Correct: Markov E[Y]/4 = 1/k² | 0.25

Why: The relationship between the columns, not the individual numbers, is what generates the next row. Squaring the deviation maps the two-sided event onto a one-sided event of Y, exactly as claimed, and E[Y] recovers the variance 1.

Sort into buckets

Sort these by value (verified), from memory, without looking back at Verify Chebyshev really is Markov-on-Y. Telling them apart on the spot is the skill; the table is only where the answer happens to be written down.

0.0498

P(|X − 1| ≥ 2); P(Y ≥ 4), Y = (X−1)²

1.0

E[Y] (= Var = σ²)

0.25

Markov E[Y]/4 = 1/k²

g1

value (verified) is "0.0498" for P(|X − 1| ≥ 2), P(Y ≥ 4), Y = (X−1)² — that is what the table on "Verify Chebyshev really is Markov-on-Y" records, and it is the single property separating this group from the rest.

g2

value (verified) is "1.0" for E[Y] (= Var = σ²) — that is what the table on "Verify Chebyshev really is Markov-on-Y" records, and it is the single property separating this group from the rest.

g3

value (verified) is "0.25" for Markov E[Y]/4 = 1/k² — that is what the table on "Verify Chebyshev really is Markov-on-Y" records, and it is the single property separating this group from the rest.

Fill in the blanks

From Chebyshev across several k — one line has had its right-hand side removed. Put it back.

import numpy as np
rng = np.random.default_rng(0)
X = rng.exponential(1.0, 2_000_000) # mu = 1, sigma = 1
for k in [2, 3, 4]:
emp = (np.abs(X - 1.0) >= k).mean()
print(f'k=___: P(|X-1|>=___)=___ 1/k^2=___')

Why: X is what everything below it consumes, so the wrong expression here fails later and somewhere else. Chebyshev is tighter than Markov but still worst-case over all distributions with σ = 1, so a specific one beats it comfortably.

Step through it

Step through Chebyshev across several k one row at a time. What is driving the change, and what would the row after the last one be?

1. Step 1: k is 2
2. Step 2: k is 3
3. Step 3: k is 4

Predict first

A student writes this, and it looks reasonable:

To bound P(|X−μ| ≥ kσ), apply Markov directly to |X − μ|: P(|X−μ| ≥ kσ) ≤ E|X−μ| / (kσ).

It is wrong. Say what breaks — and say it before you turn the page.

Correct: This is a valid bound, but it uses the mean absolute deviation E|X−μ|, not the variance — it does NOT give the clean 1/k².

Apply Markov to the squared deviation (X − μ)², whose mean is exactly σ².

Why: This is a valid bound, but it uses the mean absolute deviation E|X−μ|, not the variance — it does NOT give the clean 1/k². You've thrown away the σ² cancellation that makes Chebyshev memorable and distribution-free in k.

Fill in the blanks

Hoeffding vs the coin's real tail, with the scaffolding fading: two lines are gone now — fill both.

import numpy as np
rng = np.random.default_rng(0)
p, n, t = 0.3, 100, 0.1
means = rng.binomial(n, p, size=200_000) / n # each is Xbar of n flips
emp = (np.abs(means - p) >= t).mean()
bound = *2np.exp(-2nt2)
print('empirical P(|Xbar-p|>=t):', round(emp, 4))
print('Hoeffding 2exp(-2 n t^2):', round(bound, 4))

Why: Reproducing these unaided, rather than reading them, is what tells you the method has transferred. The bound holds with margin. It is looser than the truth (Hoeffding is distribution-free over all [0,1] variables) but far tighter than what Markov or Chebyshev would give here.

Discussion prompt

Work backwards. The example finished here:

Empirical 0.0302 sits under the Hoeffding ceiling 0.2707

What was it asked to do, and what must it have been given? Reconstruct the problem from its answer.

Hint: Every quantity in the result had to enter somewhere. Account for each one.

Fill in the blanks

The exponential collapse in n, with the scaffolding fading: two lines are gone now — fill both.

import numpy as np
t = 0.1
for n in [50, 100, 200, 500, 1000]:
bound = *2np.exp(-2nt2)
print(f'n=___: 2exp(-2 n t^2) = ___')

Why: Reproducing these unaided, rather than reading them, is what tells you the method has transferred. Each extra sample multiplies the bound by exp(−2t²) < 1, so the tail decays geometrically in n.

Step through it

Step through The exponential collapse in n one row at a time. What is driving the change, and what would the row after the last one be?

1. Step 1: n is 50
2. Step 2: n is 100
3. Step 3: n is 200
4. Step 4: n is 500
5. Step 5: n is 1000

Put in order

Put the moves of Invert Hoeffding for sample complexity into the order they have to happen.

1. Demand the failure probability be ≤ δ
2. Divide by 2 and take logs
3. Multiply by −1 (flip the inequality), divide by 2ε²

Why: These are the moves of the worked example in the order it makes them, and each one is set up by the one before it. We want P(|X̄ − E[X]| ≥ ε) ≤ δ.

Annotate

From Invert Hoeffding for sample complexity — read this one piece at a time. What is each part doing?

On: \( \boxed{\,n \ge \dfrac{\ln(2/\delta)}{2\varepsilon^2}\,} \)

• We want P(|X̄ − E[X]| ≥ ε) ≤ δ. Hoeffding's ceiling is 2e^{−2nε²}, so it suffices to force that ceiling below δ.
• ln is increasing, so it preserves ≤; ln of the exponential returns its exponent.
• Multiplying by a negative flips ≥ to the sample-size condition. Dividing by the positive 2ε² keeps direction.

Fill in the blanks

From Sample complexity, computed and checked — one line has had its right-hand side removed. Put it back.

import numpy as np
eps, delta = 0.05, 0.05
n_req = np.log(2/delta) / (2*eps2)
n = int(np.ceil(n_req))**
print('ln(2/delta) :', round(np.log(2/delta), 4))
print('n >= ... :', round(n_req, 2), '-> ceil', n)
print('check bound at n :', round(2np.exp(-2neps*2), 4), '<= 0.05')

Why: n is what everything below it consumes, so the wrong expression here fails later and somewhere else. The ceiling formula demands 738 flips to promise ±0.05 accuracy 95% of the time.

Predict first

A student writes this, and it looks reasonable:

The coin's X̄ obeys 2e^{−2nt²}, so use the same bound for the average of n draws from Exponential(1).

It is wrong. Say what breaks — and say it before you turn the page.

Correct: Hoeffding REQUIRES each variable lie in a bounded range [a,b] — the exponent is really 2nt²/(b−a)².

Match the tool to the support. For an unbounded variable with known variance, use Chebyshev on the average.

Why: Hoeffding REQUIRES each variable lie in a bounded range [a,b] — the exponent is really 2nt²/(b−a)². The exponential is unbounded (b = ∞), so (b−a)² = ∞ makes the exponent 0 and the bound the useless '≤ 2'. The clean {0,1} version silently assumed b−a = 1.

Discussion prompt

The rule this trap just fixed:

Averaging shrinks variance to σ²/n, so Chebyshev still gives a valid 1/n bound with no boundedness needed. Hoeffding's exponential speed is a reward for boundedness — you cannot claim it for free.

Now break it on purpose. Build a case that violates it and follow the consequences until something visibly fails. Where does the failure first show up — and would you have noticed it if you had not been looking?

Hint: The dangerous rules are the ones whose violation still produces an answer. If yours fails loudly, try to find one that fails quietly.

Sort into buckets

Every line below is part of the definition of one-sided vs two-sided or of shatter — one or the other, never both. Put each where it belongs.

one-sided vs two-sided

Markov bounds only the RIGHT tail P(X ≥ a) of a non-negative variable.; To bound how far X falls on BOTH sides of its mean, we need the deviation itself; that is the job of Chebyshev.

shatter

A hypothesis class H shatters a set of m points if, for EVERY one of the 2^m ways to label them +/−, some hypothesis in H realizes that exact labeling.; If even one labeling is impossible, H does not shatter the set.

b1

Markov bounds only the RIGHT tail P(X ≥ a) of a non-negative variable. To bound how far X falls on BOTH sides of its mean, we need the deviation itself — that is the job of Chebyshev.

b2

A hypothesis class H shatters a set of m points if, for EVERY one of the 2^m ways to label them +/−, some hypothesis in H realizes that exact labeling. If even one labeling is impossible, H does not shatter the set.

Predict first

Simplest class: a 1-D threshold h(x) = sign(x − θ) (with an optional flip). Two points x = 1, 2 have 2² = 4 labelings. We brute-force over thresholds and check all four are realizable:

Commit before you compute: what does A threshold shatters 2 points on a line come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: All 4 labelings realizable → the threshold class shatters 2 points

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. Sliding θ between, below, or above the points, plus the sign flip, reaches every labeling.

Step through it

Step through A threshold shatters 2 points on a line one row at a time. One of these columns never changes — find it, and say why it cannot.

1. Step 1: labeling of (x=1, x=2) is (−1, −1)
2. Step 2: labeling of (x=1, x=2) is (−1, +1)
3. Step 3: labeling of (x=1, x=2) is (+1, −1)
4. Step 4: labeling of (x=1, x=2) is (+1, +1)

Sort into buckets

Sort these by realizable?, from memory, without looking back at A threshold FAILS on 3 points. Telling them apart on the spot is the skill; the table is only where the answer happens to be written down.

6 of these work

(+, +, +) / (−, −, −) / …

impossible

(+, −, +); (−, +, −)

6 of 8

total realizable

g1

realizable? is "6 of these work" for (+, +, +) / (−, −, −) / … — that is what the table on "A threshold FAILS on 3 points" records, and it is the single property separating this group from the rest.

g2

realizable? is "impossible" for (+, −, +), (−, +, −) — that is what the table on "A threshold FAILS on 3 points" records, and it is the single property separating this group from the rest.

g3

realizable? is "6 of 8" for total realizable — that is what the table on "A threshold FAILS on 3 points" records, and it is the single property separating this group from the rest.

Fill in the blanks

From A line CANNOT shatter 4 points — XOR — one line has had its right-hand side removed. Put it back.

import numpy as np
X = np.array([[0.,0.],[1.,1.],[1.,0.],[0.,1.]])
y = np.array([0, 0, 1, 1]) # XOR: diagonal pairs share a class
g = np.linspace(-3, 3, 61)
best = 0.0
for a in g:
for b in g:
for c in g:
pred = ((aX[:,0] + bX[:,1] + c) > 0).astype(int)
best = max(best, (pred == y).mean(), (1-pred == y).mean())
print('best line accuracy on XOR:', best)

Why: g is what everything below it consumes, so the wrong expression here fails later and somewhere else. No line separates the two diagonal classes, so the (+,−,+,−)-style XOR labeling is unrealizable.

Comparison matrix

From A line CANNOT shatter 4 points — XOR: refill the best line accuracy (verified) column from what you know. The rest of the table is as it appeared.

Predict first

A student writes this, and it looks reasonable:

A linear classifier in ℝ² has 2 parameters of direction, so its VC dimension is 2.

It is wrong. Say what breaks — and say it before you turn the page.

Correct: This drops the bias term. A line through the ORIGIN (no offset) shatters only 2 points, but a general line has a third degree of freedom — its offset c — and shatters 3.

Count the offset. A general line a·x + b·y + c has d + 1 = 3 free parameters.

Why: This drops the bias term. A line through the ORIGIN (no offset) shatters only 2 points, but a general line has a third degree of freedom — its offset c — and shatters 3. Our brute force realized all 8 labelings of 3 points, so 2 is too small.

Predict first

The table runs: 3 | 8 | 8 · 4 | 15 | 16 · 5 | 26 | 32

In Watch the growth go sub-exponential, given the rows so far: what is the next one — the row where m is 10?

Correct: 10 | 176 | 1024

Why: The relationship between the columns, not the individual numbers, is what generates the next row. Up to the VC dimension the class realizes every labeling (8 = 2³).

Step through it

Step through Watch the growth go sub-exponential one row at a time. What is driving the change, and what would the row after the last one be?

1. Step 1: m is 3
2. Step 2: m is 4
3. Step 3: m is 5
4. Step 4: m is 10

Discussion prompt

Explain The VC generalization bound to a student a year behind you. No notation, no jargon they have not met — and it still has to be true.

Hint: If your explanation needs a symbol they have never seen, you are describing the notation rather than the idea.

Discussion prompt

Exam-style: n = 1000 examples, a model of VC dimension 50. Evaluate the gap term √(VC·ln n / n), and sweep to see how data and capacity move it. Runnable:

What do you need to know — or decide — before the first line can be written? List everything the problem has to hand you.

Hint: Anything you would have to invent to get started is a thing the problem must supply.

Put in order

These are the steps of The concentration → generalization pipeline, scrambled. Put them back in order before the next slide shows you.

1. Bound one hypothesis: X̄ (train err) concentrates near E[X] (true err) by Hoeffding 2e^{−2nt²}
2. Measure the class: capacity = VC dimension = largest shattered set; linear in ℝᵈ is d+1
3. Tame the search: Sauer turns the 2ᵐ labelings into a polynomial (em/d)ᵈ once m > VC
4. Union-bound the whole class: gap ≤ √(VC·ln n / n) — shrinks with n, grows with VC
5. Get n: invert to n ≥ ln(2/δ)/(2ε²) per hypothesis; capacity multiplies it

Why: This is the order the recipe itself gives. Recalling the sequence without the slide in front of you is the difference between recognising the method and being able to run it — most of what goes wrong in practice is a step done out of turn.

Discussion prompt

The concentration → generalization pipeline works on the cases you have just seen. Push it to the edge: what is the most degenerate input it still handles — empty, zero, one item, everything equal — and what is the first case where it stops being true? Name the case, not just "it breaks".

Hint: Try the smallest legal input, then the largest, then the one where two things collide. Methods are specified at their edges; the middle takes care of itself.

Eliminate the wrong options

Which inequality needs ONLY that X ≥ 0 and E[X] exists — no variance, no boundedness?

3 of these 4 are wrong. Strike them one at a time, and say what rules each one out before you strike the next. The survivor is the answer.

• A. Markov
• B. Chebyshev
• C. Hoeffding
• D. The VC generalization bound

Survives elimination: A

Why: Markov's derivation used only a·1[X≥a] ≤ X, which needs X ≥ 0 and a finite mean — nothing else. That minimal assumption set is exactly why it is the loosest of the three and serves as the lemma for the others.

Check your understanding

Which inequality needs ONLY that X ≥ 0 and E[X] exists — no variance, no boundedness?

• A. Markov (correct)
• B. Chebyshev
• C. Hoeffding
• D. The VC generalization bound

Answer: A

Why: Markov's derivation used only a·1[X≥a] ≤ X, which needs X ≥ 0 and a finite mean — nothing else. That minimal assumption set is exactly why it is the loosest of the three and serves as the lemma for the others.

Why B tempts people

Chebyshev additionally needs the variance σ²: it is literally Markov applied to (X−μ)², whose mean is Var(X). No variance, no Chebyshev.

Why C tempts people

Hoeffding requires the variables to be BOUNDED (here in [0,1]) and i.i.d. — much more structure than Markov, which is what buys its exponential tail.

Why D tempts people

The VC bound is about the capacity of a whole hypothesis class across n samples, not the tail of a single non-negative random variable.

Predict first

Chebyshev's P(|X−μ| ≥ kσ) ≤ 1/k² is obtained by applying Markov to which non-negative variable?

Answer it in your own words, now, with nothing to choose from. The options are on the next slide — and picking the right one off a list is an easier skill than producing it.

Correct: (X − μ)², at level (kσ)²

Why: Squaring the deviation makes it non-negative (so Markov applies) and gives it mean E[(X−μ)²] = σ². Markov at level (kσ)² then yields σ²/(k²σ²) = 1/k² after the variance cancels.

Check your understanding

Chebyshev's P(|X−μ| ≥ kσ) ≤ 1/k² is obtained by applying Markov to which non-negative variable?

• A. (X − μ)², at level (kσ)² (correct)
• B. X itself, at level kσ
• C. |X|, at level kσ
• D. e^{X}, at level e^{kσ}

Answer: A

Why: Squaring the deviation makes it non-negative (so Markov applies) and gives it mean E[(X−μ)²] = σ². Markov at level (kσ)² then yields σ²/(k²σ²) = 1/k² after the variance cancels.

Why B tempts people

Markov on X gives a one-sided right-tail bound E[X]/(kσ), not the two-sided deviation bound — and X need not even be centered or non-negative.

Why C tempts people

Markov on |X| bounds P(|X| ≥ kσ), a statement about magnitude around 0, not deviation |X−μ| around the mean; and its mean is E|X|, not σ².

Why D tempts people

Applying Markov to e^{X} is the Chernoff/Hoeffding route (the moment-generating function), not the Chebyshev derivation, which uses the second moment (X−μ)².

Predict first

The VC dimension of a linear classifier in ℝ² (a general line with offset) is:

Answer it in your own words, now, with nothing to choose from. The options are on the next slide — and picking the right one off a list is an easier skill than producing it.

Correct: 3 — it shatters some 3 points but no 4 (XOR fails)

Why: VC = d + 1 = 3. Our brute force realized all 8 labelings of a triangle (shatters 3) but capped at 0.75 accuracy on the 4-point XOR (cannot shatter 4). The largest shattered set has size 3.

Check your understanding

The VC dimension of a linear classifier in ℝ² (a general line with offset) is:

• A. 3 — it shatters some 3 points but no 4 (XOR fails) (correct)
• B. 2 — one per input dimension
• C. 4 — it can label any 4 points
• D. infinite — lines are very flexible

Answer: A

Why: VC = d + 1 = 3. Our brute force realized all 8 labelings of a triangle (shatters 3) but capped at 0.75 accuracy on the 4-point XOR (cannot shatter 4). The largest shattered set has size 3.

Why B tempts people

That drops the '+1' bias term. A line through the origin shatters only 2, but a general line's offset c adds a degree of freedom, letting it shatter 3.

Why C tempts people

A line cannot shatter 4 points — XOR is the counterexample, where the best any line achieves is 3 of 4 correct (0.75), never all 8 labelings.

Why D tempts people

Linear classifiers have FINITE VC dimension (d+1). Infinite VC belongs to far richer classes, e.g. 1-D thresholds on a cleverly ordered set, or sine classifiers.

Eliminate the wrong options

By the VC bound gap ≈ √(VC·ln n / n), the train–true gap shrinks when:

3 of these 4 are wrong. Strike them one at a time, and say what rules each one out before you strike the next. The survivor is the answer.

• A. n grows, or VC dimension falls
• B. VC dimension grows
• C. the learning rate increases
• D. the training error decreases

Survives elimination: A

Why: VC sits in the numerator and n dominates the denominator, so more data shrinks the gap and more capacity widens it. Our sweep confirmed it: VC=50 gap fell from 0.59 at n=1000 to 0.076 at n=100000.

Check your understanding

By the VC bound gap ≈ √(VC·ln n / n), the train–true gap shrinks when:

• A. n grows, or VC dimension falls (correct)
• B. VC dimension grows
• C. the learning rate increases
• D. the training error decreases

Answer: A

Why: VC sits in the numerator and n dominates the denominator, so more data shrinks the gap and more capacity widens it. Our sweep confirmed it: VC=50 gap fell from 0.59 at n=1000 to 0.076 at n=100000.

Why B tempts people

Higher VC is MORE capacity to overfit, so it WIDENS the bound — at VC=1000, n=1000 the term exceeds 2.6 and becomes vacuous.

Why C tempts people

The learning rate is an optimization hyperparameter; it does not appear anywhere in a capacity-vs-data generalization bound.

Why D tempts people

Lower training error can come WITH a larger gap (classic overfitting). The bound's gap term depends on VC and n, not on the training error itself.

Predict first

Fit a least-norm linear model on random tanh features, sweeping the feature count p past the sample size n = 40. Watch test MSE spike at p ≈ n then descend. Runnable as written:

Commit before you compute: what does Double descent, from a real fit come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: Test MSE dives, spikes near p ≈ n = 40, then descends again

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. At p = 40 the model exactly interpolates the 40 training points and test error explodes (~38); push p to 400 and the least-norm solution becomes smooth again, MSE back near 0.75.

Discussion prompt

Work backwards. The example finished here:

Test MSE dives, spikes near p ≈ n = 40, then descends again

What was it asked to do, and what must it have been given? Reconstruct the problem from its answer.

Hint: Every quantity in the result had to enter somewhere. Account for each one.

Predict first

A student writes this, and it looks reasonable:

The p = 40 fit had test MSE 37.9, so adding still more features must push test error even higher.

It is wrong. Say what breaks — and say it before you turn the page.

Correct: That reads only the FIRST descent and the spike.

Recognize the second descent: past the interpolation threshold, more capacity plus an implicit smoothness bias lowers test error again.

Why: That reads only the FIRST descent and the spike. Our own sweep refutes it: at p = 60 the MSE is already back down to 4.84, and at p = 400 it is 0.75 — an order of magnitude BELOW the p = 40 spike. Test error is non-monotonic in capacity.

Sort into buckets

Some of these hold up and some are the exact mistakes this lesson is built to prevent. Sort them.

Holds up

X̄ is our estimate of p, and in ML it plays the role of training error estimating true error. The whole lesson asks: how far can X̄ stray from p?; Check both cases: if X ≥ a the left side is a and a ≤ X; if X < a the left side is 0 ≤ X (since X ≥ 0). Either way it holds. Now just take expectations.; We want a two-sided bound on |X − μ|. The variable |X − μ| can be positive from either side, so we square it: (X − μ)² is non-negative — precisely what Markov needs.

Breaks

Markov gives P(X ≥ 4) ≤ 0.25, so roughly a quarter of the mass sits above 4.; To bound P(|X−μ| ≥ kσ), apply Markov directly to |X − μ|: P(|X−μ| ≥ kσ) ≤ E|X−μ| / (kσ).

sound

These are stated as this lesson states them — each one survives the edge cases Lesson 34: Concentration Inequalities & Generalization puts it through.

flawed

Each of these is lifted from a trap in this deck: reasonable-sounding, and wrong in a way that only shows up once you rely on it.

Discussion prompt

Explain Why over-parameterization can help by analogy to something with no Machine Learning in it at all — a queue, a recipe, a map, a bank balance, whatever fits. Then say where your analogy breaks.

Hint: An analogy that never breaks is not an analogy, it is the same idea wearing a hat. Find the seam — that is the part that is actually new.

Discussion prompt

Rebuild the lesson's four pillars yourself: two empirical tail checks, one sample-complexity inversion, and one capacity computation. You've derived every piece — now assemble it.

That is stated as though it always holds. Do one of two things: produce a case where it fails, or say precisely what rules such a case out. "It just does" is not on the menu.

Hint: Hunt at the extremes first — zero, one, negative, empty, equal. If every extreme survives, the reason they survive is the proof.

Comparison matrix

From Milestone 4 — the VC gap term: every row here is a choice with a cost. Fill the n column, then say which row you would actually pick and what you give up for it.

Comparison matrix

From The full program: refill the value (verified) column from what you know. The rest of the table is as it appeared.

Draw it

One page, no notation unless you need it: draw how these connect — The question generalization answers · Markov — from one indicator · Chebyshev — Markov on (X−μ)² · Hoeffding — the exponential engine · VC dimension — measuring capacity · The generalization bound. Put an arrow wherever one of them is what makes another possible, and label the arrow with why.