Lesson 32: Bias-Variance & Cross-Validation

1. Bias-Variance & Cross-Validation

Title

USAAIO · Lesson 32 · Week 11

Why a more powerful model can score worse. We derive MSE = Bias² + Variance + Noise line by line, watch the U-shaped test error emerge from real numbers, then build the cross-validation that estimates it honestly — from scratch, matched to sklearn.

2. By the end of this lesson you can

Objectives

1. Derive MSE = Bias² + Variance + Noise from E[(y − f̂)²] with every cross term shown to vanish — no hand-waving
2. Verify the decomposition numerically on a toy estimator, then measure bias² and variance empirically across polynomial degree
3. Explain the U-shaped test error: bias falls, variance rises, total dips in the middle — and diagnose underfit vs overfit from train/val curves
4. Build k-fold cross-validation from scratch, trace all k folds, and prove it matches sklearn.cross_val_score
5. Pick a CV strategy (stratified, LOO, time-series) and avoid leakage with nested CV

3. What survived from t-SNE & UMAP?

Warm-up

4. Two ways to be wrong

Section

Part 1 of 9 — the setup

5. The running example: a noisy curve

Concept

One dataset carries the whole lesson. The true signal is f(x) = sin(1.5x) on [0, 4]. We never see f; we only see noisy samples y = f(x) + ε, where ε is mean-zero noise with σ = 0.3.

\[ y = f(x) + \varepsilon, \qquad f(x) = \sin(1.5x), \qquad \varepsilon \sim \mathcal{N}(0,\, \sigma^2),\ \sigma = 0.3 \]

Our job: fit a polynomial f̂ to the samples that predicts well on new points. The only knob is the polynomial degree — that is our model complexity.

6. Break it if you can: The running example: a noisy curve

Counterexample

7. Guess the shape of the answer: One sample of the running data

Estimation

8. One sample of the running data

Worked example

Take five fixed x values, read off the true signal f(x), then add one draw of noise to get the y we actually observe. Runnable as written:

import numpy as np
np.random.seed(0)
x = np.array([0.5, 1.0, 2.0, 3.0, 3.5])
f = np.sin(1.5 * x)             # true signal (unseen in practice)
noise = 0.3 * np.random.randn(5)
y = f + noise                   # what we actually observe
print(np.round(f, 4))
print(np.round(y, 4))

9. Fill in: f(x) = sin(1.5x) for One sample of the running data

Comparison

10. The model is a random variable

Intuition

Here is the mental shift the whole lesson rests on. Because the training y carry random noise, a different training sample would give a different fitted model f̂. So f̂ is not one fixed function — it is a random object that changes every time you resample the data.

That randomness is why we cannot judge a model by one fit. We must ask two separate questions: on average, is f̂ aimed at the truth? And how much does it jump around from one dataset to the next?

Those two questions are exactly bias and variance. Everything else today is making them precise.

11. Bias and variance, in words

Concept

bias — How far the AVERAGE model f̄ = E[f̂] sits from the truth f. A too-simple model (a straight line for a curve) is aimed wrong no matter how much data you give it — that is high bias, or underfitting.

variance — How much f̂ wobbles around its own average f̄ as the training set changes. A too-flexible model chases the noise, so each dataset yields a wildly different fit — that is high variance, or overfitting.

noise (irreducible error) — The variance σ² of ε itself. Even the perfect model f can't predict ε, so σ² is a floor no model can beat.

12. Take the definitions apart: bias vs variance

Definition probe

13. Picture it first: The dartboard picture

Picture it

14. The dartboard picture

Intuition

Think of each trained model as one throw at a dartboard whose bullseye is the truth f. Retraining on a fresh dataset is another throw.

15. By analogy: The dartboard picture

Analogy

16. Derive the decomposition

Section

Part 2 of 9 — every step

17. The quantity we decompose

Concept

Fix one test input x. Its true label is the random y = f + ε. Our model, trained on a random dataset, predicts f̂. The expected squared error at x averages over BOTH sources of randomness: the noise ε AND the training set (which is what makes f̂ random).

\[ \operatorname{Err}(x) = \mathbb{E}\!\left[(y - \hat f)^2\right], \qquad y = f + \varepsilon \]

We will rewrite this single expectation as three clean pieces. The trick is one line: add and subtract the average model f̄ = E[f̂].

18. The three assumptions we use

Concept

1. E[ε] = 0 — the noise is mean-zero (no systematic offset in the labels).
2. Var(ε) = σ² — the noise has a fixed variance, and E[ε²] = σ².
3. ε is independent of the training data, hence independent of f̂.

Also write f̄ = E[f̂] for the average prediction over training sets. That is all we need — the rest is algebra.

19. Teach it back: The three assumptions we use

Explain it

20. Bias and variance, as formulas

Concept

Before the algebra, pin down the two quantities precisely so the derivation lands on named objects, not vibes. Bias is the gap between the average model and the truth; variance is the mean-square wobble of the model about its own average:

\[ \operatorname{Bias}(\hat f) = \bar f - f, \qquad \operatorname{Var}(\hat f) = \mathbb{E}\!\left[(\hat f - \bar f)^2\right] \]

21. Say it in words: Bias and variance, as formulas

Translation

\( \operatorname{Bias}(\hat f) = \bar f - f, \qquad \operatorname{Var}(\hat f) = \mathbb{E}\!\left[(\hat f - \bar f)^2\right] \)

22. What has to happen first: Step 1 — split off the noise

Ranking

23. Step 1 — split off the noise

Worked example

\[ \mathbb{E}\!\left[(y - \hat f)^2\right] = \mathbb{E}\!\left[\big((f - \hat f) + \varepsilon\big)^2\right] \]

\[ = \mathbb{E}\!\left[(f - \hat f)^2\right] + 2\,\mathbb{E}\!\left[(f - \hat f)\,\varepsilon\right] + \mathbb{E}\!\left[\varepsilon^2\right] \]

\[ = \underbrace{\mathbb{E}\!\left[(f - \hat f)^2\right]}_{\text{model error}} + \underbrace{\sigma^2}_{\text{noise}} \]

24. Decode the notation: Step 1 — split off the noise

Notation

25. What has to be given first: Step 2 — add and subtract the mean model

Missing information

26. Step 2 — add and subtract the mean model

Worked example

Now crack open the model-error term E[(f − f̂)²]. Insert f̄ = E[f̂] by adding and subtracting it — the move that separates bias from variance:

\[ \mathbb{E}\!\left[(f - \hat f)^2\right] = \mathbb{E}\!\left[\big((f - \bar f) + (\bar f - \hat f)\big)^2\right] \]

\[ = \mathbb{E}\!\left[(f - \bar f)^2\right] + 2\,\mathbb{E}\!\left[(f - \bar f)(\bar f - \hat f)\right] + \mathbb{E}\!\left[(\bar f - \hat f)^2\right] \]

27. Work backwards from the answer: Step 2 — add and subtract the mean model

Reverse engineer

28. What has to happen first: Step 3 — the last cross term also vanishes

Ranking

29. Step 3 — the last cross term also vanishes

Worked example

\[ 2\,\mathbb{E}\!\left[(f - \bar f)(\bar f - \hat f)\right] = 2\,(f - \bar f)\,\mathbb{E}\!\left[\bar f - \hat f\right] \]

\[ = 2\,(f - \bar f)\cdot 0 = 0 \]

\[ \mathbb{E}\!\left[(f - \hat f)^2\right] = \underbrace{(f - \bar f)^2}_{\text{Bias}^2} + \underbrace{\mathbb{E}\!\left[(\bar f - \hat f)^2\right]}_{\text{Variance}} \]

30. Decode the notation: Step 3 — the last cross term also vanishes

Notation

31. Assemble: the decomposition

Worked example

Put Step 1 (noise split off) together with Step 3 (bias/variance split). Every cross term was shown to be exactly zero, so nothing is hidden:

\[ \boxed{\;\mathbb{E}\!\left[(y - \hat f)^2\right] = \underbrace{(f - \bar f)^2}_{\text{Bias}^2} + \underbrace{\mathbb{E}\!\left[(\hat f - \bar f)^2\right]}_{\text{Variance}} + \underbrace{\sigma^2}_{\text{Noise}}\;} \]

32. Something is wrong here: which cross term survives?

Anomaly

33. Trap: which cross term survives?

Trap

34. Verify it numerically

Section

Part 3 of 9 — the identity is real

35. A toy where we know the answer

Intuition

Before the messy polynomials, pin the identity down on a case simple enough to check by hand. The truth is a constant f = μ = 2, and the noise has σ = 1.

Our estimator: draw n = 3 noisy points and predict their average. By symmetry this estimator is unbiased — on average it hits μ — so Bias² = 0. And the variance of an average of 3 independent draws is σ²/3 = 1/3. So the identity predicts MSE = 0 + 1/3 + 1 = 1.333.

36. Predict the next row: The unbiased averaging estimator

Pattern

37. The unbiased averaging estimator

Worked example

Simulate 200,000 retrainings. Each retraining draws a fresh 3-point dataset and predicts its mean; we also draw a fresh test label each trial. Measure bias², variance, and the raw MSE, then compare to 0 + 1/3 + 1:

import numpy as np
np.random.seed(0)
mu, sigma, n = 2.0, 1.0, 3
def fhat():
    return (mu + sigma * np.random.randn(n)).mean()   # predict the mean
T = 200000
preds = np.array([fhat() for _ in range(T)])
ynew  = mu + sigma * np.random.randn(T)                # fresh test label
mse   = np.mean((ynew - preds) ** 2)
bias2 = (preds.mean() - mu) ** 2
var   = preds.var()
print(round(bias2, 4), round(var, 4), sigma**2)
print('sum  =', round(bias2 + var + sigma**2, 4))
print('mse  =', round(mse, 4))

38. What each one costs: The unbiased averaging estimator

Trade off

39. Restore the missing line: A biased estimator, same truth

Fill the middle

40. A biased estimator, same truth

Worked example

Now break the aim on purpose: ignore the data and always predict the constant c = 1.5. The truth is still μ = 2, so Bias = 1.5 − 2 = −0.5 and Bias² = 0.25. Variance is 0 (the prediction never changes). Identity predicts 0.25 + 0 + 1 = 1.25:

import numpy as np
np.random.seed(1)
mu, sigma = 2.0, 1.0
c = 1.5                     # constant predictor, ignores the data
T = 200000
ynew = mu + sigma * np.random.randn(T)
mse  = np.mean((ynew - c) ** 2)
bias2 = (c - mu) ** 2
print('bias2 =', bias2, ' var = 0  sigma2 =', sigma**2)
print('sum   =', bias2 + 0 + sigma**2)
print('mse   =', round(mse, 4))

41. Where does each piece belong: Lesson 32: Bias-Variance & Cross-Validation

Sorting

42. Bias & variance vs complexity

Section

Part 4 of 9 — the U-curve appears

43. How complexity moves the two terms

Concept

Back to f(x) = sin(1.5x). The degree of the fitted polynomial is our complexity knob. A low degree can't bend enough to follow the sine — high bias. A high degree bends through every noisy point — high variance.

To measure this we resample: draw many training sets, fit each, and look at how the predictions behave at a grid of test points. The average prediction reveals bias; the spread reveals variance.

44. Finish it with less help: The measurement machine

Faded example

45. The measurement machine

Worked example

For a fixed degree, draw 500 training sets of 12 points each, fit a polynomial to each, and predict on a fixed test grid. P is a (500 × grid) array of predictions. Bias² is (mean over datasets − truth)²; variance is the spread across datasets — both averaged over the grid:

import numpy as np
np.random.seed(0)
f = lambda t: np.sin(1.5 * t)
sigma = 0.3
xt = np.linspace(0.2, 3.8, 50); ft = f(xt)      # test grid + truth
def preds_grid(deg, n=12):
    x = np.random.uniform(0, 4, n)
    y = f(x) + sigma * np.random.randn(n)
    return np.polyval(np.polyfit(x, y, deg), xt)
for deg in [1, 2, 3, 5, 9]:
    P = np.array([preds_grid(deg) for _ in range(500)])
    bias2 = ((P.mean(0) - ft) ** 2).mean()
    var   = P.var(0).mean()
    print(deg, round(bias2, 4), round(var, 4), round(bias2 + var + sigma**2, 4))

46. Reading the table: the U-curve

Concept

Follow the total column: 0.271 → 0.383 → 0.184 → 14.4 → huge. It dips to a minimum at degree 3, then blows up. That dip is the famous U-shaped total error — you overshoot the sweet spot in either direction.

47. More data cures variance, not bias

Worked example

Which term does more data attack? Hold the degree fixed at 5 (the high-variance regime) and grow the training-set size n = 12 → 30 → 100, remeasuring bias² and variance each time:

import numpy as np
np.random.seed(0)
f = lambda t: np.sin(1.5 * t)
sigma = 0.3
xt = np.linspace(0.2, 3.8, 50); ft = f(xt)
def preds(deg, n):
    x = np.random.uniform(0, 4, n)
    y = f(x) + sigma * np.random.randn(n)
    return np.polyval(np.polyfit(x, y, deg), xt)
for n in [12, 30, 100]:
    P = np.array([preds(5, n) for _ in range(400)])
    bias2 = ((P.mean(0) - ft) ** 2).mean()
    var   = P.var(0).mean()
    print(n, round(bias2, 4), round(var, 4))

48. Watch it run: More data cures variance, not bias

Pattern

49. Restore the missing line: Underfit and overfit, one fit each

Fill the middle

50. Underfit and overfit, one fit each

Worked example

Zoom to a single training realization and predict at one test point x0 = 2 (truth f(2) = 0.1411). Degree 1 can't reach; degree 4 is closer — but its wobble across datasets is what the table already flagged:

import numpy as np
np.random.seed(0)
x = np.array([0.5, 1.0, 2.0, 3.0, 3.5])
f = np.sin(1.5 * x)
y = f + 0.3 * np.random.randn(5)
x0 = 2.0
p1 = np.polyfit(x, y, 1)
p4 = np.polyfit(x, y, 4)
print('deg1 pred @2 =', round(np.polyval(p1, x0), 4))
print('deg4 pred @2 =', round(np.polyval(p4, x0), 4))
print('truth  f(2)  =', round(np.sin(1.5 * 2.0), 4))

51. Diagnose from curves

Section

Part 5 of 9 — train vs test

52. Guess the shape of the answer: Train error vs test error

Estimation

53. Train error vs test error

Worked example

The catch: in practice you can't compute bias and variance — they need the unknown f. What you CAN compute is training error and held-out (test) error. Fit degrees 1..12 on one 15-point training set and score both:

import numpy as np
np.random.seed(1)
f = lambda t: np.sin(1.5 * t)
sigma = 0.3
xtr = np.sort(np.random.uniform(0, 4, 15))
ytr = f(xtr) + sigma * np.random.randn(15)
xte = np.sort(np.random.uniform(0, 4, 200))
yte = f(xte) + sigma * np.random.randn(200)
for deg in [1, 2, 3, 5, 9, 12]:
    c = np.polyfit(xtr, ytr, deg)
    tr = np.mean((np.polyval(c, xtr) - ytr) ** 2)
    te = np.mean((np.polyval(c, xte) - yte) ** 2)
    print(deg, round(tr, 4), round(te, 4))

54. Watch it run: Train error vs test error

Pattern

55. Learning curves tell you which regime

Concept

Plot error against training-set size for a fixed model. High bias: train and validation error both plateau high and close together — more data won't help, you need a richer model.

High variance: a large gap — low train error, much higher validation error — that shrinks as you add data. More data (or regularization) is the cure. The gap between the two curves is your single best overfitting gauge.

56. Something is wrong here: a more complex model is always better

Anomaly

57. Trap: a more complex model is always better

Trap

58. Break it on purpose: a more complex model is always better

Break the constraint

59. Nuance: the classical U isn't the whole story

Concept

The U-curve is the classical picture, and it governs the models on this exam: polynomials, ridge, trees, k-NN. But olympiad problems sometimes probe the modern twist — double descent.

In heavily over-parameterized models (huge neural nets), test error rises to a peak near the interpolation threshold — where the model exactly fits the training data — then, surprisingly, falls again as capacity grows further. So 'more complex is worse' holds until interpolation and can reverse beyond it.

The safe takeaway is unchanged: never choose complexity by training error — let held-out error, wherever its true minimum lies, decide.

60. Cross-validation

Section

Part 6 of 9 — build k-fold

61. Why not just one split?

Concept

A single train/test split wastes data (the test points never train the model) and gives a noisy score — change which points land in the test set and the number jumps around, exactly the variance problem we've been studying.

k-fold cross-validation fixes both: chop the data into k equal folds, hold out each fold once as validation while training on the other k−1, and average the k scores. Every point is validated exactly once, so no data is wasted and the estimate has lower variance.

62. Picture it first: The rotation, drawn

Picture it

63. The rotation, drawn

Intuition

Picture 5 folds. Each row is one CV iteration: one fold (shaded) is held out to score, the rest train. Rotate the held-out fold through all 5 positions, then average the 5 scores.

64. k-fold from scratch — the split

Worked example

Start on the diabetes dataset (442 rows) so the folds and numbers are concrete. np.array_split cuts the index range into 5 nearly equal folds. Predict the sizes before printing:

import numpy as np
from sklearn.datasets import load_diabetes
X, y = load_diabetes(return_X_y=True)
folds = np.array_split(np.arange(len(X)), 5)
print('n =', len(X))
print('fold sizes =', [len(fld) for fld in folds])

65. k-fold from scratch — the loop, per fold

Worked example

For each fold i: hold it out as test, concatenate the rest as train, fit Ridge(alpha=1.0), and score. Print every fold's train size, test size, and R² — the full trace:

import numpy as np
from sklearn.linear_model import Ridge
from sklearn.datasets import load_diabetes
X, y = load_diabetes(return_X_y=True)
folds = np.array_split(np.arange(len(X)), 5)
for i in range(5):
    te = folds[i]
    tr = np.concatenate([folds[j] for j in range(5) if j != i])
    m = Ridge(alpha=1.0).fit(X[tr], y[tr])
    print(i, len(tr), len(te), round(m.score(X[te], y[te]), 4))

66. What happens as it grows: k-fold from scratch — the loop, per fold

Scale up

67. Finish it with less help: k-fold from scratch — average & verify

Faded example

68. k-fold from scratch — average & verify

Worked example

Wrap the loop in a function that returns the mean of the 5 scores, then compare to sklearn.cross_val_score with KFold(5, shuffle=False) — the same no-shuffle split we built:

import numpy as np
from sklearn.linear_model import Ridge
from sklearn.datasets import load_diabetes
from sklearn.model_selection import cross_val_score, KFold
X, y = load_diabetes(return_X_y=True)
def kfold(model, X, y, k=5):
    folds = np.array_split(np.arange(len(X)), k); s = []
    for i in range(k):
        te = folds[i]
        tr = np.concatenate([folds[j] for j in range(k) if j != i])
        model.fit(X[tr], y[tr]); s.append(model.score(X[te], y[te]))
    return np.mean(s)
mine = kfold(Ridge(alpha=1.0), X, y)
skl  = cross_val_score(Ridge(alpha=1.0), X, y, cv=KFold(5, shuffle=False)).mean()
print(round(mine, 4), round(skl, 4), np.isclose(mine, skl))

69. Something is wrong here: shuffle mismatch breaks the comparison

Anomaly

70. Trap: shuffle mismatch breaks the comparison

Trap

71. Which of these survive contact with Lesson 32: Bias-Variance & Cross-Validation?

Two truths and a lie

72. The CV strategy zoo

Section

Part 7 of 9 — pick the right split

73. Four splitters, four situations

Concept

Plain k-fold assumes rows are exchangeable. When they are not — imbalanced classes, or time order — you must switch splitter or the estimate lies.

74. Fill in: use when for Four splitters, four situations

Comparison

75. Restore the missing line: Stratified preserves class balance

Fill the middle

76. Stratified preserves class balance

Worked example

Take a 90/10 imbalanced label. Plain k-fold splits randomly, so some folds get too few positives; stratified k-fold forces every fold to the global 10% rate. Watch the per-fold positive rate:

import numpy as np
from sklearn.model_selection import KFold, StratifiedKFold
y = np.array([0]*90 + [1]*10)            # 90/10 imbalance
np.random.seed(0)
y = y[np.random.permutation(len(y))]
def pos_rates(splitter):
    return [round(y[te].mean(), 3) for _, te in splitter.split(np.zeros_like(y), y)]
print('plain ', pos_rates(KFold(5, shuffle=True, random_state=0)))
print('strat ', pos_rates(StratifiedKFold(5, shuffle=True, random_state=0)))

77. Predict the next row: Time-series: never train on the future

Pattern

78. Time-series: never train on the future

Worked example

For ordered data, a random fold would let the model peek at future points to predict the past — leakage. TimeSeriesSplit only ever trains on earlier indices and tests on later ones, growing the training window:

import numpy as np
from sklearn.model_selection import TimeSeriesSplit
idx = np.arange(10)
for tr, te in TimeSeriesSplit(n_splits=4).split(idx):
    print('train', list(tr), 'test', list(te))

79. Watch it run: Time-series: never train on the future

Pattern

80. What has to be given first: Leave-one-out: k = n

Missing information

81. Leave-one-out: k = n

Worked example

The extreme of k-fold: set k = n, so each fold is a single point. That is leave-one-out (LOO) — n fits, each training on all but one point. It wrings the most training data out of a tiny dataset, at the cost of n model fits. Compare its MSE to 5-fold on diabetes:

import numpy as np
from sklearn.linear_model import Ridge
from sklearn.datasets import load_diabetes
from sklearn.model_selection import cross_val_score, LeaveOneOut, KFold
X, y = load_diabetes(return_X_y=True)
loo = cross_val_score(Ridge(1.0), X, y, cv=LeaveOneOut(),
                      scoring='neg_mean_squared_error')
kf  = cross_val_score(Ridge(1.0), X, y, cv=KFold(5, shuffle=False),
                      scoring='neg_mean_squared_error')
print('LOO folds =', len(loo))
print('LOO  MSE  =', round(-loo.mean(), 2))
print('5-fold MSE=', round(-kf.mean(), 2))

82. Tuning without cheating

Section

Part 8 of 9 — nested CV

83. The tuning trap in one sentence

Concept

If you try many hyperparameters and report the best score on the same data you used to pick them, that score is optimistically biased — you fit the evaluation set through model selection. This is a form of data leakage.

The cure is separation: tune on inner folds, evaluate on outer folds the tuning never touched. That is nested CV — CV inside CV.

84. Guess the shape of the answer: Cheating vs nested CV, measured

Estimation

85. Cheating vs nested CV, measured

Worked example

Grid over Ridge alpha. The cheating protocol picks the alpha that maximizes a single CV and reports that same CV. Nested CV tunes alpha in an inner loop and reports the untouched outer loop. Compare the two estimates:

import numpy as np
from sklearn.linear_model import Ridge
from sklearn.datasets import load_diabetes
from sklearn.model_selection import GridSearchCV, cross_val_score, KFold
X, y = load_diabetes(return_X_y=True)
grid = {'alpha': [0.01, 0.1, 1.0, 10.0, 100.0]}
inner = KFold(5, shuffle=True, random_state=0)
outer = KFold(5, shuffle=True, random_state=0)
gs = GridSearchCV(Ridge(), grid, cv=inner)
nested = cross_val_score(gs, X, y, cv=outer).mean()
best = max(grid['alpha'], key=lambda a: cross_val_score(Ridge(alpha=a), X, y, cv=outer).mean())
cheat = cross_val_score(Ridge(alpha=best), X, y, cv=outer).mean()
print('cheat:', best, round(cheat, 4))
print('nested:', round(nested, 4))

86. Work backwards from the answer: Cheating vs nested CV, measured

Reverse engineer

87. Searching the hyperparameter space

Concept

• Grid search — try every combination on a grid; exhaustive but blows up with dimensions.
• Random search — sample combinations at random; often finds a good setting far faster when few hyperparameters matter.
• Bayesian optimization — model the score surface and probe where improvement is likely; most sample-efficient, most machinery.

Whatever the search, it belongs in the inner loop of nested CV. The outer loop's only job is to report a number you can trust.

88. Teach it back: Searching the hyperparameter space

Explain it

89. A hyperparameter IS a bias-variance dial

Concept

The reason tuning matters is that a hyperparameter like ridge's λ slides you along the same bias-variance curve. Larger λ shrinks the weights toward zero — less flexibility, less variance, more bias. Smaller λ does the reverse.

\[ \lambda \uparrow \;\Longrightarrow\; \text{variance} \downarrow,\ \text{bias} \uparrow \qquad\qquad \lambda \downarrow \;\Longrightarrow\; \text{variance} \uparrow,\ \text{bias} \downarrow \]

So choosing λ by CV is just finding the bottom of the U in a different coordinate. Polynomial degree, tree depth, k in k-NN, λ in ridge — every complexity knob is a bias-variance dial you set with held-out error.

90. Pattern, checks & project

Section

Part 9 of 9

91. Rebuild the recipe: The bias-variance / CV toolkit

Ranking

92. The bias-variance / CV toolkit

Pattern

1. Decompose: every test error is Bias² + Variance + Noise — the cross terms vanish in expectation
2. Diagnose: high train & high test error = bias (underfit); low train, high test = variance (overfit)
3. Pick complexity at the minimum of held-out error (bottom of the U), never on training error
4. Estimate held-out error with k-fold CV — stratified for imbalance, time-series for temporal data, LOO for tiny n
5. Tune hyperparameters in the inner loop of nested CV so the outer estimate stays unbiased

93. Where does it stop working: The bias-variance / CV toolkit

Edge cases

94. Rule out three: Check yourself — the decomposition

Elimination

95. Check yourself — the decomposition

Check

Match the symptom to the term. Think about train vs test.

96. Answer it before you see the options: Check yourself — the cross term

Prediction

97. Check yourself — the cross term

Check

Recall why the decomposition has exactly three terms.

98. Answer it before you see the options: Check yourself — k-fold

Prediction

99. Check yourself — k-fold

Check

How does 5-fold use each data point?

100. Rule out three: Check yourself — nested CV

Elimination

101. Check yourself — nested CV

Check

Why two loops instead of one?

102. Project: choose the polynomial degree by CV

Concept

Bring both halves together. Using only NumPy, build k-fold CV on the sin(1.5x) data and use it to pick the polynomial degree — recovering the sweet spot the bias-variance table predicted, without ever peeking at the true f.

Build rules: type every line yourself, seed the RNG so your numbers match, and read errors instead of deleting them — a shape mismatch usually means a fold index went wrong.

103. By analogy: Project: choose the polynomial degree by CV

Analogy

104. Milestone 1 — the data

Worked example

Your turn: generate 30 noisy points from f(x) = sin(1.5x) on [0, 4] with σ = 0.3 and seed = 2. Predict x's shape and y's rough range before printing.

Hint: np.sort(np.random.uniform(0, 4, 30)) for x; add 0.3 * np.random.randn(30) to sin(1.5x) for y.

import numpy as np
np.random.seed(2)
f = lambda t: np.sin(1.5 * t)
x = np.sort(np.random.uniform(0, 4, 30))
y = f(x) + 0.3 * np.random.randn(30)
print(x.shape, round(y.min(), 3), round(y.max(), 3))

105. Watch it run: Milestone 1 — the data

Pattern

106. Milestone 2 — CV MSE for one degree

Worked example

Your turn: write cv_mse(deg, k=5) — split indices into k folds, fit np.polyfit on k−1, score MSE on the held-out fold, average. Predict roughly the degree-3 CV MSE (recall test error bottomed near degree 3).

Hint: folds = np.array_split(np.arange(len(x)), k); tr = np.concatenate([folds[j] for j in range(k) if j != i]); error is np.mean((np.polyval(c, x[te]) - y[te])**2).

import numpy as np
np.random.seed(2)
f = lambda t: np.sin(1.5 * t)
x = np.sort(np.random.uniform(0, 4, 30))
y = f(x) + 0.3 * np.random.randn(30)
def cv_mse(deg, k=5):
    folds = np.array_split(np.arange(len(x)), k); errs = []
    for i in range(k):
        te = folds[i]
        tr = np.concatenate([folds[j] for j in range(k) if j != i])
        c = np.polyfit(x[tr], y[tr], deg)
        errs.append(np.mean((np.polyval(c, x[te]) - y[te]) ** 2))
    return np.mean(errs)
print(round(cv_mse(3), 4))

107. What each one costs: Milestone 2 — CV MSE for one degree

Trade off

108. Milestone 3 — sweep and pick

Worked example

Your turn: run cv_mse over degrees 1, 2, 3, 5, 9 and pick the degree with the smallest CV MSE. Predict which one wins before you print.

Hint: build a dict {deg: cv_mse(deg)}, then min(scores, key=scores.get).

import numpy as np
np.random.seed(2)
f = lambda t: np.sin(1.5 * t)
x = np.sort(np.random.uniform(0, 4, 30))
y = f(x) + 0.3 * np.random.randn(30)
def cv_mse(deg, k=5):
    folds = np.array_split(np.arange(len(x)), k); errs = []
    for i in range(k):
        te = folds[i]
        tr = np.concatenate([folds[j] for j in range(k) if j != i])
        c = np.polyfit(x[tr], y[tr], deg)
        errs.append(np.mean((np.polyval(c, x[te]) - y[te]) ** 2))
    return np.mean(errs)
scores = {d: round(cv_mse(d), 4) for d in [1, 2, 3, 5, 9]}
print(scores)
print('best degree =', min(scores, key=scores.get))

109. Fill in: CV MSE for Milestone 3 — sweep and pick

Comparison

110. The full program

Concept

import numpy as np
np.random.seed(2)
f = lambda t: np.sin(1.5 * t)
x = np.sort(np.random.uniform(0, 4, 30))     # 1. noisy data
y = f(x) + 0.3 * np.random.randn(30)

def cv_mse(deg, k=5):                          # 2. k-fold CV MSE
    folds = np.array_split(np.arange(len(x)), k); errs = []
    for i in range(k):
        te = folds[i]
        tr = np.concatenate([folds[j] for j in range(k) if j != i])
        c = np.polyfit(x[tr], y[tr], deg)
        errs.append(np.mean((np.polyval(c, x[te]) - y[te]) ** 2))
    return np.mean(errs)

scores = {d: round(cv_mse(d), 4) for d in [1, 2, 3, 5, 9]}  # 3. sweep
print(scores)
print('best degree =', min(scores, key=scores.get))

CV chose degree 3 — the same sweet spot the bias-variance table found — using only the noisy data. You picked model complexity honestly, without ever seeing the truth.

111. Break it if you can: The full program

Counterexample

112. Show it off

Concept

Slides closed, out loud: explain (1) the three terms of MSE = Bias² + Variance + Noise and why the two cross terms vanish, (2) why training error falls monotonically but test error is U-shaped, and (3) why nested CV's outer loop must never see the tuning.

Stretch: swap cv_mse to use StratifiedKFold on a classification target, then wrap the degree sweep in an outer loop to make it a true nested CV. Bias-variance underlies regularization (Week 12) and ensembles (Week 23) — bagging attacks variance, boosting attacks bias.

113. Connect it up: Lesson 32: Bias-Variance & Cross-Validation

Connect it up

114. What you can do now

Recap

• Derive MSE = Bias² + Variance + Noise from E[(y − f̂)²], showing both cross terms vanish — and verify it numerically (1.33 and 1.25)
• Measure bias² and variance empirically across complexity and read the U-shaped total error (sweet spot at degree 3 for our sin data)
• Diagnose underfit (high train & test error) vs overfit (low train, high test) from train/val and learning curves
• Build k-fold CV from scratch, trace all 5 folds, and match sklearn (0.4102) — with stratified / LOO / time-series variants
• Tune with nested CV so the honest estimate (0.4858) isn't inflated by leakage (0.4894)