Lesson 6: Gradient Intuition & PyTorch Autograd

Match the pairs

From The road ahead — match each one to what it actually does. The descriptions have been shuffled.

• c1. 1. The gradient
• c2. 3. Identities
• c3. 4. Autograd
• c4. 5. Build it

• b1. Partials → a steepest-ascent vector.
• b2. 2w, x, 2Ax — the ML three.
• b3. The chain rule, automated.
• b4. Gradient descent from scratch.

Why: 1. The gradient, 3. Identities, 4. Autograd, 5. Build it are easy to tell apart while they are sitting next to their descriptions and much harder afterwards, which is what this checks.

Discussion prompt

One function carries the whole first half of this lesson. It takes two inputs and returns one number — a surface floating over the (x, y) plane.

That is stated as though it always holds. Do one of two things: produce a case where it fails, or say precisely what rules such a case out. "It just does" is not on the menu.

Hint: Hunt at the extremes first — zero, one, negative, empty, equal. If every extreme survives, the reason they survive is the proof.

Discussion prompt

Explain What a partial derivative means by analogy to something with no Machine Learning in it at all — a queue, a recipe, a map, a bank balance, whatever fits. Then say where your analogy breaks.

Hint: An analogy that never breaks is not an analogy, it is the same idea wearing a hat. Find the seam — that is the part that is actually new.

Discussion prompt

Explain Freezing y is slicing the surface to a student a year behind you. No notation, no jargon they have not met — and it still has to be true.

Hint: If your explanation needs a symbol they have never seen, you are describing the notation rather than the idea.

Put in order

Put the moves of Partial in x, term by term into the order they have to happen.

1. Term x² → 2x
2. Term 3xy → 3y
3. Add the three → ∂f/∂x = 2x + 3y

Why: These are the moves of the worked example in the order it makes them, and each one is set up by the one before it. Ordinary power rule in x: d(x²)/dx = 2x.

Annotate

From Partial in x, term by term — read this one piece at a time. What is each part doing?

On: \( \frac{\partial f}{\partial x} = 2x + 3y \)

• Ordinary power rule in x: d(x²)/dx = 2x. No y here to worry about.
• 3y is a constant coefficient on x, so d(3y·x)/dx = 3y. The x differentiates away, the frozen y stays.
• y² has no x in it at all, so as a function of x it is a constant — and the derivative of a constant is 0.

Discussion prompt

Partial in y, term by term — before any calculation: what is the plan? Name the moves in order, in plain English, without doing the arithmetic.

Hint: It starts with: Term x² → 0

Discussion prompt

Work backwards. The example finished here:

Add the three → ∂f/∂y = 3x + 2y

What was it asked to do, and what must it have been given? Reconstruct the problem from its answer.

Hint: Every quantity in the result had to enter somewhere. Account for each one.

Predict first

The gradient is a formula in x and y. To get an actual arrow, plug in a point. Use (1, 2):

Commit before you compute: what does Evaluate the gradient at (1, 2) come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: ∇f(1,2) = [8, 7]

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. The arrow at (1,2). Both components positive, so f increases as you move in +x and +y from this point.

Comparison matrix

From Evaluate the gradient at (1, 2): refill the formula column from what you know. The rest of the table is as it appeared.

Discussion prompt

A second gradient, with a mixed term — before any calculation: what is the plan? Name the moves in order, in plain English, without doing the arithmetic.

Hint: It starts with: ∂g/∂x: freeze y, differentiate x²y + y³ in x → 2xy + 0 = 2xy

Comparison matrix

From A second gradient, with a mixed term: every row here is a choice with a cost. Fill the formula column, then say which row you would actually pick and what you give up for it.

Sort into buckets

Sort these by ‖∇f‖ (steepness), from memory, without looking back at One arrow per point: a vector field. Telling them apart on the spot is the skill; the table is only where the answer happens to be written down.

0.000

(0, 0)

3.606

(1, 0); (0, 1)

10.630

(1, 2); (2, 1)

g1

‖∇f‖ (steepness) is "0.000" for (0, 0) — that is what the table on "One arrow per point: a vector field" records, and it is the single property separating this group from the rest.

g2

‖∇f‖ (steepness) is "3.606" for (1, 0), (0, 1) — that is what the table on "One arrow per point: a vector field" records, and it is the single property separating this group from the rest.

g3

‖∇f‖ (steepness) is "10.630" for (1, 2), (2, 1) — that is what the table on "One arrow per point: a vector field" records, and it is the single property separating this group from the rest.

Predict first

A student writes this, and it looks reasonable:

Goal: minimize the loss, so the gradient must point downhill toward the minimum — follow it.

It is wrong. Say what breaks — and say it before you turn the page.

Correct: Treats the gradient as a downhill arrow and walks ALONG it.

The gradient points toward steepest ascent — uphill, toward larger loss. To go down, negate it.

Why: Treats the gradient as a downhill arrow and walks ALONG it. But ∇f is steepest ASCENT, so this climbs toward LARGER loss — the loss goes up every step and training diverges.

Predict first

The table runs: ∇f at (1,2) | [8, 7] · v̂ = v/‖v‖ | [0.7071, 0.7071] · D_v f = ∇f · v̂ | 10.6066

In Directional derivative along v = [1, 1], given the rows so far: what is the next one — the row where quantity is ‖∇f‖ (the maximum)?

Correct: ‖∇f‖ (the maximum) | 10.6301

Why: The relationship between the columns, not the individual numbers, is what generates the next row. Divide each component by ‖v‖ = √2 to get a unit vector.

Draw it

Translate both ways. First write the expression above as a sentence with no symbols in it at all. Then cover it, and write your sentence back as notation. If the two versions disagree, the disagreement is the thing to fix.

Put in order

Put the moves of Descent direction has slope −‖∇f‖ into the order they have to happen.

1. Descent unit vector: v̂ = −∇f / ‖∇f‖ = [−8, −7]/√113
2. D_v f = ∇f · v̂ = −(8² + 7²)/√113 = −113/√113 = −√113
3. Compare to the level-curve direction

Why: These are the moves of the worked example in the order it makes them, and each one is set up by the one before it. Point exactly opposite the gradient, normalized to unit length so we read a pure slope.

Discussion prompt

‖w‖² is the sum of squared components. Differentiate with respect to one component wₖ, then reassemble the vector:

What do you need to know — or decide — before the first line can be written? List everything the problem has to hand you.

Hint: Anything you would have to invent to get started is a thing the problem must supply.

Annotate

Walk the callouts on Identity 1: ∇‖w‖² = 2w, derived. Each one is a place this is easy to get subtly wrong.

• The squared norm is just the sum of each coordinate squared — no vectors needed to differentiate it.
• Only the j = k term contains wₖ; every other term is constant in wₖ and drops. That one term gives 2wₖ.
• Component k of the gradient is 2wₖ, so the whole gradient vector is 2w. This is the vector version of d(x²)/dx = 2x.

Discussion prompt

Identity 2: ∇(wᵀx) = x, derived — before any calculation: what is the plan? Name the moves in order, in plain English, without doing the arithmetic.

Hint: It starts with: Write it out: wᵀx = Σⱼ wⱼ xⱼ

Annotate

Walk the callouts on Identity 2: ∇(wᵀx) = x, derived. Each one is a place this is easy to get subtly wrong.

• A dot product is a sum of pairwise products. Here x is the constant, w is the variable.
• Only the j = k term has wₖ; its derivative is the constant coefficient xₖ. Everything else drops.
• Component k is xₖ, so the gradient is the vector x itself — the vector version of d(bx)/dx = b.

Predict first

Identity 3: ∇(xᵀAx), the quadratic form is about to be worked. State your hypothesis first: which rule or definition decides this one, and what is the first move it forces? Then watch whether the example agrees with you.

Correct: Partial in xₖ hits the i = k copy and the j = k copy

Why: In the double sum, xₖ shows up wherever i = k (giving Σⱼ Aₖⱼ xⱼ = row k of Ax) and wherever j = k (giving Σᵢ xᵢ Aᵢₖ = row k of Aᵀx).

A hypothesis you wrote down is falsifiable; a vague sense of how it will go is not. If the example opens somewhere else, that gap is the thing worth chasing.

Annotate

From Identity 3: ∇(xᵀAx), the quadratic form — read this one piece at a time. What is each part doing?

On: \( A = A^\top \;\Longrightarrow\; \frac{d\,(x^\top A x)}{dx} = 2Ax \)

• In the double sum, xₖ shows up wherever i = k (giving Σⱼ Aₖⱼ xⱼ = row k of Ax) and wherever j = k (giving Σᵢ xᵢ Aᵢₖ = row k of Aᵀx).
• One term from each occurrence of x. This is the fully general answer — no symmetry assumed yet.
• Then A + Aᵀ = 2A. Loss Hessians and XᵀX are symmetric, so 2Ax is the form you meet in practice.

Discussion prompt

All three identities in one real loss — before any calculation: what is the plan? Name the moves in order, in plain English, without doing the arithmetic.

Hint: It starts with: Quadratic term wᵀ(XᵀX)w → 2XᵀXw

Sort into buckets

Sort these by value at w = [0.5, 1.0], from memory, without looking back at All three identities in one real loss. Telling them apart on the spot is the skill; the table is only where the answer happens to be written down.

[−0.5, −1.5, −1.5]

residual Xw − y

4.75

L = ‖Xw − y‖²

[−7.0, −16.0]

hand: 2Xᵀ(Xw − y); autograd w.grad

g1

value at w = [0.5, 1.0] is "[−0.5, −1.5, −1.5]" for residual Xw − y — that is what the table on "All three identities in one real loss" records, and it is the single property separating this group from the rest.

g2

value at w = [0.5, 1.0] is "4.75" for L = ‖Xw − y‖² — that is what the table on "All three identities in one real loss" records, and it is the single property separating this group from the rest.

g3

value at w = [0.5, 1.0] is "[−7.0, −16.0]" for hand: 2Xᵀ(Xw − y), autograd w.grad — that is what the table on "All three identities in one real loss" records, and it is the single property separating this group from the rest.

Predict first

A student writes this, and it looks reasonable:

Differentiate xᵀAx. By analogy with the linear term wᵀx → x, guess the gradient is just Ax.

It is wrong. Say what breaks — and say it before you turn the page.

Correct: Misses that x appears TWICE in the quadratic form.

x sits on both sides of A, so the product rule yields two terms: Ax + Aᵀx.

Why: Misses that x appears TWICE in the quadratic form. The product rule gives a contribution from EACH copy, so you are short by exactly the Aᵀx term — off by a factor near 2.

Discussion prompt

The rule this trap just fixed:

Verified: A = [[2,1],[1,3]], x = [1,2] → 2Ax = [8, 14], matching autograd exactly. Guessing Ax would give [4, 7] — half the true gradient, and training would crawl at half speed.

Now break it on purpose. Build a case that violates it and follow the consequences until something visibly fails. Where does the failure first show up — and would you have noticed it if you had not been looking?

Hint: The dangerous rules are the ones whose violation still produces an answer. If yours fails loudly, try to find one that fails quietly.

Discussion prompt

Read the picture before the words. What is this showing, and what is the one thing it is built to make obvious? Commit to an answer, then read on.

Hint: Name the parts, then say what changes between them — and if nothing changes, say what is being held still.

Fill in the blanks

From Autograd reproduces our hand gradient — one line has had its right-hand side removed. Put it back.

import torch
x = torch.tensor(1.0, requires_grad=True)
y = torch.tensor(2.0, requires_grad=True)
f = x2 + 3xy + y2
f.backward()
print(f.item(), x.grad.item(), y.grad.item())

Why: f is what everything below it consumes, so the wrong expression here fails later and somewhere else. The scalar f triggers a single reverse traversal; each leaf receives its partial derivative.

Put in order

Put the moves of The chain rule autograd runs, by hand into the order they have to happen.

1. Inner: u = x² + 1, so du/dx = 2x
2. Outer: f = u³, so df/du = 3u²
3. Chain them: df/dx = (df/du)(du/dx) = 3u² · 2x = 75 · 4 = 300

Why: These are the moves of the worked example in the order it makes them, and each one is set up by the one before it. The first recorded node. At x=2, u = 4 + 1 = 5 and du/dx = 4.

Comparison matrix

From The chain rule autograd runs, by hand: refill the hand value column from what you know. The rest of the table is as it appeared.

Fill in the blanks

From Descending a loss bowl with autograd — one line has had its right-hand side removed. Put it back.

import torch
w = torch.tensor(0.0, requires_grad=True)
b = torch.tensor(0.0, requires_grad=True)
lr = 0.1
for step in range(5):
loss = (w-3)2 + (b+1)2
loss.backward()
with torch.no_grad():
w -= lr * w.grad
b -= lr * b.grad
w.grad.zero_(); b.grad.zero_()

Why: loss is what everything below it consumes, so the wrong expression here fails later and somewhere else. This four-line rhythm is identical for a 2-parameter bowl and a 70-billion-parameter model.

Step through it

Step through Descending a loss bowl with autograd one row at a time. What is driving the change, and what would the row after the last one be?

1. Step 1: step is 1
2. Step 2: step is 2
3. Step 3: step is 3
4. Step 4: step is 4
5. Step 5: step is 5

Discussion prompt

Explain Why the update lives under no_grad() to a student a year behind you. No notation, no jargon they have not met — and it still has to be true.

Hint: If your explanation needs a symbol they have never seen, you are describing the notation rather than the idea.

Predict first

Try the update in place without the guard, then with it. The first raises; the second works:

Commit before you compute: what does The no_grad error, live come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: The unguarded in-place update raises RuntimeError

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. PyTorch forbids in-place edits of a leaf that requires grad, because it would corrupt the recorded graph.

Predict first

A student writes this, and it looks reasonable:

.grad holds this step's gradient, so just call .backward() each loop and read it — no cleanup needed.

It is wrong. Say what breaks — and say it before you turn the page.

Correct: PyTorch ACCUMULATES (adds) into .grad.

.grad is a running sum — you must reset it to zero every iteration.

Why: PyTorch ACCUMULATES (adds) into .grad. The true gradient is −6.0, but the stale −6.0 from the previous call was summed on top, giving −12.0. Steps get bigger every iteration and training silently corrupts.

Sort into buckets

Some of these hold up and some are the exact mistakes this lesson is built to prevent. Sort them.

Holds up

Five parts, each building on the last. We start from a single slope and end holding a working optimizer you wrote yourself.; One function carries the whole first half of this lesson. It takes two inputs and returns one number — a surface floating over the (x, y) plane.; The partial derivative ∂f/∂x is the slope you feel walking in the +x direction only — y held fixed, as if it were a constant.

Breaks

Goal: minimize the loss, so the gradient must point downhill toward the minimum — follow it.; Differentiate xᵀAx. By analogy with the linear term wᵀx → x, guess the gradient is just Ax.

sound

These are stated as this lesson states them — each one survives the edge cases Lesson 6: Gradient Intuition & PyTorch Autograd puts it through.

flawed

Each of these is lifted from a trap in this deck: reasonable-sounding, and wrong in a way that only shows up once you rely on it.

Put in order

These are the steps of The autograd training loop — the recipe, scrambled. Put them back in order before the next slide shows you.

1. Create parameters with requires_grad=True
2. Forward: compute the scalar loss from the parameters
3. Backward: loss.backward() fills every .grad
4. Update under torch.no_grad(): θ -= lr * θ.grad
5. Zero the gradients: θ.grad.zero_() — or the next step's gradient is corrupted
6. Repeat until the loss stops improving

Why: This is the order the recipe itself gives. Recalling the sequence without the slide in front of you is the difference between recognising the method and being able to run it — most of what goes wrong in practice is a step done out of turn.

Discussion prompt

The autograd training loop — the recipe works on the cases you have just seen. Push it to the edge: what is the most degenerate input it still handles — empty, zero, one item, everything equal — and what is the first case where it stops being true? Name the case, not just "it breaks".

Hint: Try the smallest legal input, then the largest, then the one where two things collide. Methods are specified at their edges; the middle takes care of itself.

Eliminate the wrong options

For f(x, y) = x²y + y³, what is ∂f/∂x at (2, 3)?

3 of these 4 are wrong. Strike them one at a time, and say what rules each one out before you strike the next. The survivor is the answer.

• A. 12
• B. 31
• C. 4
• D. 27

Survives elimination: A

Why: Treat y as a constant: ∂f/∂x = 2xy. At (2, 3): 2·2·3 = 12. The y³ term has no x, so it drops to 0.

Check your understanding

For f(x, y) = x²y + y³, what is ∂f/∂x at (2, 3)?

• A. 12 (correct)
• B. 31
• C. 4
• D. 27

Answer: A

Why: Treat y as a constant: ∂f/∂x = 2xy. At (2, 3): 2·2·3 = 12. The y³ term has no x, so it drops to 0.

Why B tempts people

This is ∂f/∂y = x² + 3y² = 4 + 27 = 31 — differentiated with respect to the WRONG variable (y instead of x).

Why C tempts people

Treated y as if it were the constant 1 instead of its value 3, giving 2x = 4. The frozen variable keeps its actual value.

Why D tempts people

This is just the y³ term's y-derivative (3y² = 27) — differentiated the wrong term and the wrong variable.

Predict first

You are minimizing a loss L(θ). At the current θ, ∇L = [4, −2]. Which step decreases L fastest for a small fixed step size?

Answer it in your own words, now, with nothing to choose from. The options are on the next slide — and picking the right one off a list is an easier skill than producing it.

Correct: Move along [−4, 2]

Why: Steepest descent is the negative gradient: −∇L = [−4, 2]. The gradient points to steepest ASCENT, so flip its sign to go downhill fastest.

Check your understanding

You are minimizing a loss L(θ). At the current θ, ∇L = [4, −2]. Which step decreases L fastest for a small fixed step size?

• A. Move along [−4, 2] (correct)
• B. Move along [4, −2]
• C. Move along [2, 4]
• D. Move along [−2, 4]

Answer: A

Why: Steepest descent is the negative gradient: −∇L = [−4, 2]. The gradient points to steepest ASCENT, so flip its sign to go downhill fastest.

Why B tempts people

This is +∇L itself — the direction of steepest ASCENT. It increases L fastest, the exact opposite of minimization.

Why C tempts people

This is perpendicular to ∇L (dot product 4·2 + (−2)·4 = 0), so it's along a level curve — to first order it doesn't change L at all, let alone decrease it fastest.

Why D tempts people

Swapped the components of −∇L = [−4, 2] into [−2, 4]. It's neither the descent direction nor perpendicular — a plausible-looking rearrangement that points the wrong way.

Predict first

At a point, ∇f = [3, 4]. What is the directional derivative of f in the direction v = [4, 3]?

Commit to an answer, then rate it — certain, fairly sure, or guessing — and write the rating down before you turn the page.

Correct: 24/5 = 4.8

Why: Normalize v: ‖v‖ = √(16+9) = 5, so v̂ = [0.8, 0.6]. Then D_v f = ∇f · v̂ = 3·0.8 + 4·0.6 = 2.4 + 2.4 = 4.8.

The rating matters as much as the answer: confident-and-wrong is the combination that survives revision, because nothing about it feels like it needs revisiting.

Check your understanding

At a point, ∇f = [3, 4]. What is the directional derivative of f in the direction v = [4, 3]?

• A. 24/5 = 4.8 (correct)
• B. 24
• C. 5
• D. 7

Answer: A

Why: Normalize v: ‖v‖ = √(16+9) = 5, so v̂ = [0.8, 0.6]. Then D_v f = ∇f · v̂ = 3·0.8 + 4·0.6 = 2.4 + 2.4 = 4.8.

Why B tempts people

Dotted with the raw v = [4, 3] (3·4 + 4·3 = 24) WITHOUT normalizing. That mixes the slope with the length of v — you must divide by ‖v‖ = 5.

Why C tempts people

This is ‖∇f‖ = √(9+16) = 5, the MAXIMUM directional derivative (along ∇f itself). But v = [4, 3] is not aligned with ∇f = [3, 4], so the actual rate is less.

Why D tempts people

Added the components of ∇f (3 + 4 = 7) — not a dot product at all, and no normalization.

Predict first

For symmetric A = [[2, 1], [1, 3]] and x = [1, 2], what is d(xᵀAx)/dx?

Answer it in your own words, now, with nothing to choose from. The options are on the next slide — and picking the right one off a list is an easier skill than producing it.

Correct: [8, 14]

Why: For symmetric A the gradient is 2Ax. Ax = [2·1 + 1·2, 1·1 + 3·2] = [4, 7], so 2Ax = [8, 14]. Confirmed by autograd.

Check your understanding

For symmetric A = [[2, 1], [1, 3]] and x = [1, 2], what is d(xᵀAx)/dx?

• A. [8, 14] (correct)
• B. [4, 7]
• C. [3, 4]
• D. [2, 6]

Answer: A

Why: For symmetric A the gradient is 2Ax. Ax = [2·1 + 1·2, 1·1 + 3·2] = [4, 7], so 2Ax = [8, 14]. Confirmed by autograd.

Why B tempts people

This is Ax without the factor of 2 — the classic 'forgot x appears twice' mistake. The product rule doubles it to 2Ax = [8, 14].

Why C tempts people

This is Ax + x componentwise or a similar slip — it uses neither the doubling nor the full rows of A. The correct product Ax = [4, 7] uses whole rows, then doubles to [8, 14].

Why D tempts people

This is the diagonal of A applied to x ([2·1, 3·2] = [2, 6]) — ignoring the off-diagonal 1s that couple the two components. The full matrix product Ax uses entire rows, not just diagonals.

Eliminate the wrong options

w = torch.tensor(2.0, requires_grad=True). You run loss=(w-5)**2; loss.backward() TWICE in a row with no zero_grad() between. What does w.grad read?

3 of these 4 are wrong. Strike them one at a time, and say what rules each one out before you strike the next. The survivor is the answer.

• A. −12.0
• B. −6.0
• C. 0.0
• D. An error — backward can only be called once

Survives elimination: A

Why: Each backward computes ∂loss/∂w = 2(w−5) = 2(2−5) = −6, and PyTorch ADDS it into the existing .grad. Two calls → −6 + −6 = −12.0.

Check your understanding

w = torch.tensor(2.0, requires_grad=True). You run loss=(w-5)**2; loss.backward() TWICE in a row with no zero_grad() between. What does w.grad read?

• A. −12.0 (correct)
• B. −6.0
• C. 0.0
• D. An error — backward can only be called once

Answer: A

Why: Each backward computes ∂loss/∂w = 2(w−5) = 2(2−5) = −6, and PyTorch ADDS it into the existing .grad. Two calls → −6 + −6 = −12.0.

Why B tempts people

This is the gradient from a SINGLE backward pass. It's correct only if you zero_grad() between calls — which the question explicitly skips.

Why C tempts people

.grad starts at None, not 0, and it accumulates; it is never reset to zero unless you call zero_grad() yourself.

Why D tempts people

Re-running the forward line (loss=(w-5)**2) builds a fresh graph each time, so a second backward is legal here. The 'call backward once' error only appears if you reuse the SAME graph without retain_graph=True.

Discussion prompt

Explain Project: gradient descent from scratch by analogy to something with no Machine Learning in it at all — a queue, a recipe, a map, a bank balance, whatever fits. Then say where your analogy breaks.

Hint: An analogy that never breaks is not an analogy, it is the same idea wearing a hat. Find the seam — that is the part that is actually new.

Discussion prompt

Work backwards. The example finished here:

Predict then check: grad(0) = 2(0−5) = −10

What was it asked to do, and what must it have been given? Reconstruct the problem from its answer.

Hint: Every quantity in the result had to enter somewhere. Account for each one.

Predict first

The table runs: 1 | 0.000 | −10.000 | 1.0000 | 16.00000 · 2 | 1.000 | −8.000 | 1.8000 | 10.24000 · 3 | 1.800 | −6.400 | 2.4400 | 6.55360 · 4 | 2.440 | −5.120 | 2.9520 | 4.19430 · 5 | 2.952 | −4.096 | 3.3616 | 2.68435 · 6 | 3.362 | −3.277 | 3.6893 | 1.71799

In Milestone 2 — the descent loop, given the rows so far: what is the next one — the row where step is 7?

Correct: 7 | 3.689 | −2.621 | 3.9514 | 1.09951

Why: The relationship between the columns, not the individual numbers, is what generates the next row. grad(w) = 2(w−5) → 0 as w → 5, so each step is smaller than the last.

Step through it

Step through Milestone 2 — the descent loop one row at a time. What is driving the change, and what would the row after the last one be?

1. Step 1: step is 1
2. Step 2: step is 2
3. Step 3: step is 3
4. Step 4: step is 4
5. Step 5: step is 5
6. Step 6: step is 6
7. Step 7: step is 7

Discussion prompt

Your turn: at w = 2, confirm your hand gradient grad(2) equals what PyTorch computes. Predict the number first (it's 2(2−5)).

What do you need to know — or decide — before the first line can be written? List everything the problem has to hand you.

Hint: Anything you would have to invent to get started is a thing the problem must supply.

Comparison matrix

From Milestone 3 — verify against autograd: every row here is a choice with a cost. Fill the gradient at w = 2 column, then say which row you would actually pick and what you give up for it.

Predict first

All three milestones assembled into one runnable script:

Commit before you compute: what does The full program come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: Success criterion: w marches 0 → ~3.95, and the check line prints two matching −6.0

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. If your run matches the table below, you built gradient descent from the math up and proved it against PyTorch's engine.

Comparison matrix

From The full program: refill the value (verified) column from what you know. The rest of the table is as it appeared.

Discussion prompt

Slides closed, out loud: explain (1) why grad(0) is negative, (2) why the steps shrink as w nears 5, and (3) what zero_grad() would prevent if you looped the autograd version.

That is stated as though it always holds. Do one of two things: produce a case where it fails, or say precisely what rules such a case out. "It just does" is not on the menu.

Hint: Hunt at the extremes first — zero, one, negative, empty, equal. If every extreme survives, the reason they survive is the proof.

Draw it

One page, no notation unless you need it: draw how these connect — From one slope to many · The directional derivative · Three identities that power all of ML · Autograd: gradients for free · Your turn: build gradient descent. Put an arrow wherever one of them is what makes another possible, and label the arrow with why.