Lesson 40: Linear Regression & the PyTorch Training Loop

Discussion prompt

Before we open Lesson 40: Linear Regression & the PyTorch Training Loop: without looking back, what was the main idea of Mock Exam — Phase 1 Coding, and what could you do by the end of it that you could not do before?

Hint: One sentence for the idea, one for the skill. If the second one is blank, that is the part to revisit.

Comparison matrix

From The running example: refill the x column from what you know. The rest of the table is as it appeared.

Discussion prompt

Stack a ones column (for the intercept) next to x to get the design matrix X, so Xw produces all four predictions at once:

That is stated as though it always holds. Do one of two things: produce a case where it fails, or say precisely what rules such a case out. "It just does" is not on the menu.

Hint: Hunt at the extremes first — zero, one, negative, empty, equal. If every extreme survives, the reason they survive is the proof.

Discussion prompt

Explain The closed form (Lesson 7, in one line) by analogy to something with no Machine Learning in it at all — a queue, a recipe, a map, a bank balance, whatever fits. Then say where your analogy breaks.

Hint: An analogy that never breaks is not an analogy, it is the same idea wearing a hat. Find the seam — that is the part that is actually new.

Put in order

Put the moves of Build XᵀX, entry by entry into the order they have to happen.

1. Top-left = ones·ones = Σ1 = 4
2. Off-diagonal = ones·x = Σxᵢ = 1+2+3+4 = 10
3. Bottom-right = x·x = Σxᵢ² = 1+4+9+16 = 30

Why: These are the moves of the worked example in the order it makes them, and each one is set up by the one before it. Four 1s dotted with four 1s: 1+1+1+1 = 4.

Annotate

From Build XᵀX, entry by entry — read this one piece at a time. What is each part doing?

On: \( X^\top X = \begin{bmatrix} 4 & 10 \\ 10 & 30 \end{bmatrix} \)

• Four 1s dotted with four 1s: 1+1+1+1 = 4. This entry is just n, the number of points.
• The ones vector dotted with x simply sums x. It fills both off-diagonal slots by symmetry.

Draw it

Draw what Build Xᵀy, entry by entry just did — the shape of it, not the line-by-line working. One picture, labels only where you need them. Then check it against the steps: anything you could not draw is a step you followed rather than understood.

Predict first

For [[a, b], [c, d]] the inverse is 1/det · [[d, −b], [−c, a]]. First the determinant det = ad − bc:

Commit before you compute: what does Invert the 2×2 — determinant first come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: det = 20 ≠ 0 → invertible, unique solution

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. A nonzero determinant guarantees XᵀX is invertible, so w★ exists and is unique.

Discussion prompt

Work backwards. The example finished here:

det = 20 ≠ 0 → invertible, unique solution

What was it asked to do, and what must it have been given? Reconstruct the problem from its answer.

Hint: Every quantity in the result had to enter somewhere. Account for each one.

Discussion prompt

w★ = (XᵀX)⁻¹ Xᵀy. Dot each row of the inverse with [22, 64], one entry at a time:

What do you need to know — or decide — before the first line can be written? List everything the problem has to hand you.

Hint: Anything you would have to invent to get started is a thing the problem must supply.

Draw it

Translate both ways. First write the expression above as a sentence with no symbols in it at all. Then cover it, and write your sentence back as notation. If the two versions disagree, the disagreement is the thing to fix.

Predict first

The table runs: XtX | [[4.0, 10.0], [10.0, 30.0]] · Xty | [22.0, 64.0]

In Confirm w in NumPy, given the rows so far: what is the next one — the row where object is w = [w₀, w₁]?

Correct: w = [w₀, w₁] | [1.0, 1.8]

Why: The relationship between the columns, not the individual numbers, is what generates the next row. np.linalg.solve factors the 2×2 (LU) and never forms an explicit inverse — faster and more numerically stable than inv.

Comparison matrix

From Confirm w in NumPy: every row here is a choice with a cost. Fill the printed value (verified) column, then say which row you would actually pick and what you give up for it.

Step through it

Step through Predictions and residuals — full trace one row at a time. What is driving the change, and what would the row after the last one be?

1. Step 1: x is 1
2. Step 2: x is 2
3. Step 3: x is 3
4. Step 4: x is 4

Discussion prompt

Explain Why iterate when we have the exact answer? to a student a year behind you. No notation, no jargon they have not met — and it still has to be true.

Hint: If your explanation needs a symbol they have never seen, you are describing the notation rather than the idea.

Put in order

Put the moves of Derive the MSE gradient, term by term into the order they have to happen.

1. Start from L(w) = (1/n)‖Xw − y‖²
2. Differentiate the squared norm: ∇‖Xw − y‖² = 2Xᵀ(Xw − y)
3. Carry the 1/n through

Why: These are the moves of the worked example in the order it makes them, and each one is set up by the one before it. The mean squared error on the design matrix — the exact loss PyTorch's MSELoss computes.

Draw it

Draw what Derive the MSE gradient, term by term just did — the shape of it, not the line-by-line working. One picture, labels only where you need them. Then check it against the steps: anything you could not draw is a step you followed rather than understood.

Fill in the blanks

From The gradient in code, checked two ways — one line has had its right-hand side removed. Put it back.

import numpy as np
x = np.array([1., 2., 3., 4.])
y = np.array([3., 4., 7., 8.])
X = np.c_[np.ones(4), x]
n = 4
def grad(w): return (2.0/n) * (X.T @ (X @ w - y))
print("grad at [0, 0] =", grad(np.array([0., 0.])).tolist())
print("grad at [1.0, 1.8] =", grad(np.array([1.0, 1.8])).round(6).tolist())

Why: y is what everything below it consumes, so the wrong expression here fails later and somewhere else. grad([0,0]) = (2/4)·Xᵀ(−y) = 0.5·(−[22,64]) = [−11, −32]: strongly downhill.

Annotate

From Step 1 of gradient descent, by hand — read this one piece at a time. What is each part doing?

On: \( w^{(1)} = \begin{bmatrix} 0 \\ 0 \end{bmatrix} - 0.1\begin{bmatrix} -11 \\ -32 \end{bmatrix} = \begin{bmatrix} 1.1 \\ 3.2 \end{bmatrix} \)

• Subtracting a negative gradient moves the intercept UP toward its target of 1.0 — the right direction.
• The slope jumps to 3.2 — overshooting the target 1.8, because the gradient was so steep at the start. GD will correct this on the next step.

Fill in the blanks

Watch four steps descend, with the scaffolding fading: two lines are gone now — fill both.

import numpy as np
x = np.array([1., 2., 3., 4.])
y = np.array([3., 4., 7., 8.])
X = np.c_[np.ones(4), x]
n = 4
w = np.array([0., 0.]); eta = 0.1
for step in range(4):
g = (2.0/n) * (X.T @ (X @ w - y))
w = w - eta * g
loss = ((X @ w - y)2).mean()**
print(f"step ___: w = ___ loss = ___")

Why: Reproducing these unaided, rather than reading them, is what tells you the method has transferred. Each step over/undershoots the target 1.8, but the loss shrinks 15.6 → 7.1 → 3.3 → 1.6 — the bowl is convex, so downhill always makes progress even when a single coordinate zig-zags.

Predict first

A student writes this, and it looks reasonable:

Bigger steps must reach the minimum faster — crank η up to 0.5.

It is wrong. Say what breaks — and say it before you turn the page.

Correct: Each step overshoots the minimum and lands farther out than it started, so the gradient grows, the next step overshoots more, and the loss blows up.

Keep η small enough that each step actually decreases the loss — here 0.1 works.

Why: Each step overshoots the minimum and lands farther out than it started, so the gradient grows, the next step overshoots more, and the loss blows up. Too-large η diverges — the opposite of what you wanted.

Annotate

Walk the callouts on Trap: a learning rate that is too large. Each one is a place this is easy to get subtly wrong.

• Each step overshoots the minimum and lands farther out than it started, so the gradient grows, the next step overshoots more, and the loss blows up. Too-large η diverges — the opposite of what you wanted.
• A small enough step size guarantees descent on a convex loss. If your loss ever increases or turns to NaN, halve η first — it's the #1 cause of a 'broken' training run.

Predict first

nn.Linear(1, 1) holds one weight and one bias, initialized randomly. Seed first so the numbers are reproducible:

Commit before you compute: what does Step B — nn.Linear is ŷ = wx + b come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: It starts with random weight −0.0075, bias 0.5364 — a bad line

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. With torch.manual_seed(0), nn.Linear(1,1) initializes to these exact values.

Sort into buckets

These are the pieces of Lesson 40: Linear Regression & the PyTorch Training Loop, out of order. Put each one back under the part of the lesson it belongs to.

One dataset, the closed form

The running example; Best line = smallest total squared miss; The design matrix and MSE

The same w, by gradient descent

Why iterate when we have the exact answer?; The update rule; Derive the MSE gradient, term by term

The PyTorch training loop

Let the framework compute the gradient; Step A — data as tensors of shape (n, 1); Step B — nn.Linear is ŷ = wx + b

s1

One dataset, the closed form is where Lesson 40: Linear Regression & the PyTorch Training Loop puts The running example, Best line = smallest total squared miss, The design matrix and MSE. Knowing which part of the lesson a problem belongs to is most of knowing which method to reach for.

s2

The same w, by gradient descent is where Lesson 40: Linear Regression & the PyTorch Training Loop puts Why iterate when we have the exact answer?, The update rule, Derive the MSE gradient, term by term. Knowing which part of the lesson a problem belongs to is most of knowing which method to reach for.

s3

The PyTorch training loop is where Lesson 40: Linear Regression & the PyTorch Training Loop puts Let the framework compute the gradient, Step A — data as tensors of shape (n, 1), Step B — nn.Linear is ŷ = wx + b. Knowing which part of the lesson a problem belongs to is most of knowing which method to reach for.

Step through it

Step through Step C — MSELoss scores the fit one row at a time. One of these columns never changes — find it, and say why it cannot.

1. Step 1: pred is 2
2. Step 2: pred is 3
3. Step 3: pred is 6
4. Step 4: pred is 7

Fill in the blanks

From One iteration, hand-checked against autograd — one line has had its right-hand side removed. Put it back.

import torch, numpy as np
torch.manual_seed(0)
x = np.array([1., 2., 3., 4.]); y = np.array([3., 4., 7., 8.])
xt = torch.tensor(x, dtype=torch.float32).unsqueeze(1)
yt = torch.tensor(y, dtype=torch.float32).unsqueeze(1)
model = torch.nn.Linear(1, 1)
opt = torch.optim.SGD(model.parameters(), lr=0.1)
opt.zero_grad()
loss = torch.nn.MSELoss()(model(xt), yt)
loss.backward()
print("weight.grad =", round(model.weight.grad.item(), 4))
print("bias.grad =", round(model.bias.grad.item(), 4))
opt.step()
print("weight now =", round(model.weight.item(), 4))
print("bias now =", round(model.bias.item(), 4))

Why: loss is what everything below it consumes, so the wrong expression here fails later and somewhere else. SGD applied w ← w − lr·grad exactly.

Comparison matrix

From The full loop — train to convergence: refill the intercept column from what you know. The rest of the table is as it appeared.

Step through it

Step through A loss trace: watch it converge one row at a time. What is driving the change, and what would the row after the last one be?

1. Step 1: epoch is 0
2. Step 2: epoch is 1
3. Step 3: epoch is 10
4. Step 4: epoch is 100
5. Step 5: epoch is 1000

Predict first

A student writes this, and it looks reasonable:

The gradient is fresh each pass, so just backward() then step() — skip zero_grad().

It is wrong. Say what breaks — and say it before you turn the page.

Correct: PyTorch ACCUMULATES gradients: each backward() ADDS to the existing .grad.

Call opt.zero_grad() at the top of every iteration.

Why: PyTorch ACCUMULATES gradients: each backward() ADDS to the existing .grad. Without zeroing, step 2 uses grad₁+grad₂, so the optimizer takes wrong-sized steps and the loss bounces (43.30, then 49.75) instead of descending.

Annotate

Walk the callouts on Trap: forgetting zero_grad(). Each one is a place this is easy to get subtly wrong.

• PyTorch ACCUMULATES gradients: each backward() ADDS to the existing .grad. Without zeroing, step 2 uses grad₁+grad₂, so the optimizer takes wrong-sized steps and the loss bounces (43.30, then 49.75) instead of descending.
• zero_grad() resets every .grad to zero so each step uses only THIS iteration's gradient. This is the single most common training-loop bug; put it first, every loop, always.

Fill in the blanks

weight_decay shrinks the fit, with the scaffolding fading: two lines are gone now — fill both.

import torch, numpy as np
x = np.array([1., 2., 3., 4.]); y = np.array([3., 4., 7., 8.])
xt = torch.tensor(x, dtype=torch.float32).unsqueeze(1)
yt = torch.tensor(y, dtype=torch.float32).unsqueeze(1)
lossfn = torch.nn.MSELoss()
for wd in (0.0, 0.5):
torch.manual_seed(0)
m = torch.nn.Linear(1, 1)
opt = torch.optim.SGD(m.parameters(), lr=0.1, weight_decay=wd)
for _ in range(1000):
opt.zero_grad(); lossfn(m(xt), yt).backward(); opt.step()
print(f"wd=___: slope=___ intercept=___")

Why: Reproducing these unaided, rather than reading them, is what tells you the method has transferred. With weight_decay=0.5 the penalty pulls the parameters toward zero, so the fit backs off the exact OLS solution.

Fill in the blanks

From Degree-2 features on our data — one line has had its right-hand side removed. Put it back.

import numpy as np
x = np.array([1., 2., 3., 4.])
y = np.array([3., 4., 7., 8.])
Xp = np.c_[np.ones(4), x, x2] # [1, x, x^2]
w = np.linalg.solve(Xp.T@Xp, Xp.T@y)
print("w (deg 2) =", w.round(4).tolist())
print("SSE =", round(((Xp@w - y)2).sum(), 4))

Why: Xp is what everything below it consumes, so the wrong expression here fails later and somewhere else. OLS finds w = [1.0, 1.8, 0.0]: the quadratic term is exactly zero because our y = [3,4,7,8] has no curvature.

Predict first

The table runs: 1 | 2 | 0.800000 · 2 | 3 | 0.800000

In Degree 3 overfits: SSE hits zero, given the rows so far: what is the next one — the row where degree is 3?

Correct: 3 | 4 | 0.000000

Why: The relationship between the columns, not the individual numbers, is what generates the next row. 4 params for 4 points interpolates exactly: zero training error.

Predict first

A student writes this, and it looks reasonable:

Higher degree gives lower SSE, so pick the degree with the smallest training error.

It is wrong. Say what breaks — and say it before you turn the page.

Correct: Training error always drops (or ties) as you add parameters — degree 3 hits 0 by interpolating the 4 points.

Choose the degree with the smallest validation error — error on data the model did not train on.

Why: Training error always drops (or ties) as you add parameters — degree 3 hits 0 by interpolating the 4 points. Optimizing training error alone always picks the most complex, most overfit model.

Sort into buckets

Some of these hold up and some are the exact mistakes this lesson is built to prevent. Sort them.

Holds up

We keep one tiny dataset for the whole lesson so every number is checkable by hand. Four points — think of x as hours studied and y as a score:; Stack a ones column (for the intercept) next to x to get the design matrix X, so Xw produces all four predictions at once:; Setting ∇L = 0 and solving gives the normal equations — a square, solvable system whose unique answer is the least-squares w:

Breaks

Bigger steps must reach the minimum faster — crank η up to 0.5.; The gradient is fresh each pass, so just backward() then step() — skip zero_grad().

sound

These are stated as this lesson states them — each one survives the edge cases Lesson 40: Linear Regression & the PyTorch Training Loop puts it through.

flawed

Each of these is lifted from a trap in this deck: reasonable-sounding, and wrong in a way that only shows up once you rely on it.

Discussion prompt

Explain R² and RMSE to a student a year behind you. No notation, no jargon they have not met — and it still has to be true.

Hint: If your explanation needs a symbol they have never seen, you are describing the notation rather than the idea.

Discussion prompt

ȳ = 22/4 = 5.5. Deviations of y = [3,4,7,8] from 5.5 are [−2.5, −1.5, 1.5, 2.5]; squared and summed give SS_tot:

What do you need to know — or decide — before the first line can be written? List everything the problem has to hand you.

Hint: Anything you would have to invent to get started is a thing the problem must supply.

Discussion prompt

Work backwards. The example finished here:

SS_tot = 6.25 + 2.25 + 2.25 + 6.25 = 17.0

What was it asked to do, and what must it have been given? Reconstruct the problem from its answer.

Hint: Every quantity in the result had to enter somewhere. Account for each one.

Predict first

The final sanity check: fit sklearn.LinearRegression and confirm it prints the same intercept, slope, and R²:

Commit before you compute: what does Cross-check against sklearn come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: sklearn agrees to the digit: intercept 1.0, slope 1.8, R² 0.9529

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. Three independent routes — by-hand OLS, the PyTorch loop, and sklearn — all land on the identical [1.0, 1.8] with R² 0.9529.

Step through it

Step through Cross-check against sklearn one row at a time. One of these columns never changes — find it, and say why it cannot.

1. Step 1: source is by-hand OLS
2. Step 2: source is PyTorch SGD
3. Step 3: source is sklearn

Discussion prompt

Explain Don't trust R² alone — check residuals by analogy to something with no Machine Learning in it at all — a queue, a recipe, a map, a bank balance, whatever fits. Then say where your analogy breaks.

Hint: An analogy that never breaks is not an analogy, it is the same idea wearing a hat. Find the seam — that is the part that is actually new.

Put in order

These are the steps of The regression + training-loop recipe, scrambled. Put them back in order before the next slide shows you.

1. Shape the data: design matrix X = [1, features]; tensors reshaped to (n, features) for nn.Linear
2. Model + loss: nn.Linear (add polynomial columns for curves); MSELoss
3. Loop, in order: zero_grad → forward → loss → backward → step, with a small learning rate
4. Regularize: weight_decay=λ (L2 / ridge) to shrink weights; pick complexity by validation error
5. Score & check: R², RMSE, a residual plot, and match a from-scratch / sklearn baseline

Why: This is the order the recipe itself gives. Recalling the sequence without the slide in front of you is the difference between recognising the method and being able to run it — most of what goes wrong in practice is a step done out of turn.

Discussion prompt

The regression + training-loop recipe works on the cases you have just seen. Push it to the edge: what is the most degenerate input it still handles — empty, zero, one item, everything equal — and what is the first case where it stops being true? Name the case, not just "it breaks".

Hint: Try the smallest legal input, then the largest, then the one where two things collide. Methods are specified at their edges; the middle takes care of itself.

Eliminate the wrong options

What is the correct order of one PyTorch training iteration?

3 of these 4 are wrong. Strike them one at a time, and say what rules each one out before you strike the next. The survivor is the answer.

• A. zero_grad → forward → loss → backward → step
• B. forward → backward → step → zero_grad → loss
• C. backward → forward → loss → step
• D. forward → loss → step → backward

Survives elimination: A

Why: Clear old grads, run the forward pass to predict, score the loss, backprop to fill every .grad, then let the optimizer update the parameters. Skipping zero_grad accumulates gradients across steps (the earlier trap).

Check your understanding

What is the correct order of one PyTorch training iteration?

• A. zero_grad → forward → loss → backward → step (correct)
• B. forward → backward → step → zero_grad → loss
• C. backward → forward → loss → step
• D. forward → loss → step → backward

Answer: A

Why: Clear old grads, run the forward pass to predict, score the loss, backprop to fill every .grad, then let the optimizer update the parameters. Skipping zero_grad accumulates gradients across steps (the earlier trap).

Why B tempts people

backward() differentiates the loss, so it must come AFTER loss is computed — not before it. And zero_grad belongs at the START of the iteration, not after step.

Why C tempts people

You cannot call backward() before a forward pass has built the computation graph — there is nothing to differentiate yet.

Why D tempts people

step() applies the gradients, so it must come AFTER backward() computes them. Stepping before backward updates with stale (or zero) gradients.

Predict first

Linear regression has an exact closed form. Why train it with gradient descent anyway?

Answer it in your own words, now, with nothing to choose from. The options are on the next slide — and picking the right one off a list is an easier skill than producing it.

Correct: GD scales to huge data and generalizes to models with no closed form

Why: The closed form needs to solve a d×d system (O(d³), and only exists for linear least squares). GD scales to large d/n and — the whole point of learning it now — works for every later model (logistic regression, MLPs, transformers) where no closed form exists.

Check your understanding

Linear regression has an exact closed form. Why train it with gradient descent anyway?

• A. GD scales to huge data and generalizes to models with no closed form (correct)
• B. GD reaches a lower loss than the closed form
• C. The closed form is wrong when the data has noise
• D. GD does not need a loss function

Answer: A

Why: The closed form needs to solve a d×d system (O(d³), and only exists for linear least squares). GD scales to large d/n and — the whole point of learning it now — works for every later model (logistic regression, MLPs, transformers) where no closed form exists.

Why B tempts people

On convex MSE both reach the SAME global minimum — we verified GD lands on the identical [1.0, 1.8], loss 0.2. GD is not more accurate, just more general.

Why C tempts people

The closed form IS the exact least-squares fit for noisy data — that is precisely what OLS computes. Noise is not a reason to abandon it.

Why D tempts people

GD is defined by differentiating a loss — step 3 of the loop computes the loss and step 4 backprops it. There is no avoiding the loss function.

Check your understanding

For L(w) = (1/n)‖Xw − y‖², what is ∇L(w)?

• A. (2/n) Xᵀ(Xw − y) (correct)
• B. (2/n) X(Xw − y)
• C. (2/n) Xᵀ(y − Xw)
• D. (1/n) Xᵀ(Xw − y)

Answer: A

Why: Chain rule: the outer 2(Xw − y) times the inner Xᵀ, all scaled by 1/n, gives (2/n)Xᵀ(Xw − y). At w = [1.0, 1.8] this evaluates to [0, 0], confirming the minimum.

Why B tempts people

X (not Xᵀ) has shape (n×d); X·(n-vector) is undefined — the shapes don't conform. You must transpose: Xᵀ maps the n-vector residual back to a d-vector gradient.

Why C tempts people

This is the NEGATIVE gradient (y − Xw instead of Xw − y). Descending with it — w ← w − η·(this) — would move UPHILL and increase the loss.

Why D tempts people

The factor is 2/n, not 1/n: differentiating the square (·)² brings down a factor of 2 that does not cancel. Dropping it just rescales the effective learning rate, but the stated derivative is wrong.

Eliminate the wrong options

Setting weight_decay=λ in a PyTorch optimizer implements:

3 of these 4 are wrong. Strike them one at a time, and say what rules each one out before you strike the next. The survivor is the answer.

• A. L2 regularization (ridge) — a λ‖w‖² penalty that shrinks weights toward 0
• B. L1 regularization (lasso) — drives weights to exact zeros
• C. Dropout — randomly zeroing activations during training
• D. Early stopping — halting when validation loss rises

Survives elimination: A

Why: weight_decay adds the L2 penalty λ‖w‖² to the loss — exactly ridge regression (Lesson 12). It pulls parameters toward zero; in our run it shrank the intercept from 1.00 to 0.76.

Check your understanding

Setting weight_decay=λ in a PyTorch optimizer implements:

• A. L2 regularization (ridge) — a λ‖w‖² penalty that shrinks weights toward 0 (correct)
• B. L1 regularization (lasso) — drives weights to exact zeros
• C. Dropout — randomly zeroing activations during training
• D. Early stopping — halting when validation loss rises

Answer: A

Why: weight_decay adds the L2 penalty λ‖w‖² to the loss — exactly ridge regression (Lesson 12). It pulls parameters toward zero; in our run it shrank the intercept from 1.00 to 0.76.

Why B tempts people

L1/lasso (which yields exact zeros for sparsity) is NOT the built-in weight_decay. To get L1 you add λΣ|w| to the loss manually — there is no optimizer flag for it.

Why C tempts people

Dropout is a separate layer (nn.Dropout) that randomly zeroes activations; it is not an optimizer argument and does not penalize weight size.

Why D tempts people

Early stopping is a training-control policy based on a validation signal, applied outside the optimizer — not a per-step penalty like weight_decay.

Discussion prompt

Fit ŷ = w₀ + w₁·x on our data x = [1,2,3,4], y = [3,4,7,8] three ways and prove they agree, then add weight_decay. You derived every piece — now assemble it yourself.

That is stated as though it always holds. Do one of two things: produce a case where it fails, or say precisely what rules such a case out. "It just does" is not on the menu.

Hint: Hunt at the extremes first — zero, one, negative, empty, equal. If every extreme survives, the reason they survive is the proof.

Comparison matrix

From Milestone 3 — verify & regularize: every row here is a choice with a cost. Fill the slope column, then say which row you would actually pick and what you give up for it.

Comparison matrix

From The full program: refill the value (verified) column from what you know. The rest of the table is as it appeared.

Draw it

One page, no notation unless you need it: draw how these connect — One dataset, the closed form · The same w, by gradient descent · The PyTorch training loop · Regularize & bend the line · Score the fit · Your turn: train it. Put an arrow wherever one of them is what makes another possible, and label the arrow with why.