Lesson 33: Gradient Checking & Debugging

Discussion prompt

Before we open Lesson 33: Gradient Checking & Debugging: without looking back, what was the main idea of Bias-Variance & Cross-Validation, and what could you do by the end of it that you could not do before?

Hint: One sentence for the idea, one for the skill. If the second one is blank, that is the part to revisit.

Discussion prompt

Backprop is just the chain rule applied by hand (or by a framework) across many layers. One dropped factor, one flipped sign, one wrong axis — and the gradient is wrong but not crashing.

That is stated as though it always holds. Do one of two things: produce a case where it fails, or say precisely what rules such a case out. "It just does" is not on the menu.

Hint: Hunt at the extremes first — zero, one, negative, empty, equal. If every extreme survives, the reason they survive is the proof.

Discussion prompt

Explain The tool: Taylor expansion by analogy to something with no Machine Learning in it at all — a queue, a recipe, a map, a bank balance, whatever fits. Then say where your analogy breaks.

Hint: An analogy that never breaks is not an analogy, it is the same idea wearing a hat. Find the seam — that is the part that is actually new.

Put in order

Put the moves of Forward difference is only O(ε) into the order they have to happen.

1. Subtract f(x) from the (x+ε) expansion
2. Divide both sides by ε
3. The leftover error is proportional to ε → O(ε)

Why: These are the moves of the worked example in the order it makes them, and each one is set up by the one before it. Everything except the constant term survives; this is f(x+ε) − f(x).

Annotate

From Forward difference is only O(ε) — read this one piece at a time. What is each part doing?

On: \( f(x+\epsilon) - f(x) = f'(x)\,\epsilon + \tfrac{1}{2}f''(x)\,\epsilon^2 + \cdots \)

• Everything except the constant term survives; this is f(x+ε) − f(x).
• Isolate the slope estimate on the left; every remaining term keeps one factor of ε or more.
• The first uncancelled term is ½f''(x)·ε. Halving ε only halves the error. That linear decay is why forward differences are the weaker tool.

Discussion prompt

Central difference: subtract the two expansions — before any calculation: what is the plan? Name the moves in order, in plain English, without doing the arithmetic.

Hint: It starts with: Subtract: f(x+ε) − f(x−ε)

Discussion prompt

Work backwards. The example finished here:

The f'' term cancelled — that is the whole trick

What was it asked to do, and what must it have been given? Reconstruct the problem from its answer.

Hint: Every quantity in the result had to enter somewhere. Account for each one.

Comparison matrix

From Central difference is O(ε²): refill the order column from what you know. The rest of the table is as it appeared.

Predict first

Take f(x) = x³ at x = 2 (true derivative 12). Halve ε each row and watch the central error fall by exactly 4× while the forward error only halves. Runnable standalone:

Commit before you compute: what does See O(ε²) in real numbers come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: Central error ratio is 4.0 every time ε halves

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. 0.1→0.05 takes 1.0e-2 → 2.5e-3, a factor of 4 = 2².

Sort into buckets

Sort these by central ratio, from memory, without looking back at See O(ε²) in real numbers. Telling them apart on the spot is the skill; the table is only where the answer happens to be written down.

—

0.1000

4.0×

0.0500; 0.0250; 0.0125

g1

central ratio is "—" for 0.1000 — that is what the table on "See O(ε²) in real numbers" records, and it is the single property separating this group from the rest.

g2

central ratio is "4.0×" for 0.0500, 0.0250, 0.0125 — that is what the table on "See O(ε²) in real numbers" records, and it is the single property separating this group from the rest.

Discussion prompt

Sweep ε across many orders of magnitude on f(x)=x² at x=3 (true derivative 6). The error is a U: big ε = truncation, tiny ε = round-off. Standalone:

What do you need to know — or decide — before the first line can be written? List everything the problem has to hand you.

Hint: Anything you would have to invent to get started is a thing the problem must supply.

Comparison matrix

From The truncation vs round-off U-curve: every row here is a choice with a cost. Fill the central error column, then say which row you would actually pick and what you give up for it.

Predict first

A student writes this, and it looks reasonable:

Use the forward difference (f(x+ε) − f(x))/ε and make ε as tiny as possible — smaller ε must mean less error, right?

It is wrong. Say what breaks — and say it before you turn the page.

Correct: Two mistakes stacked. Forward is only O(ε), so it is inaccurate to begin with; and ε=1e-12 sits deep in the round-off regime where cancellation destroys the digits.

Use the central difference with ε ≈ 1e-5 — O(ε²) accurate and sitting at the bottom of the U-curve.

Why: Two mistakes stacked. Forward is only O(ε), so it is inaccurate to begin with; and ε=1e-12 sits deep in the round-off regime where cancellation destroys the digits. A correct gradient then 'fails' the check on noise.

Discussion prompt

Explain Why relative, not absolute, error to a student a year behind you. No notation, no jargon they have not met — and it still has to be true.

Hint: If your explanation needs a symbol they have never seen, you are describing the notation rather than the idea.

Sort into buckets

These are the pieces of Lesson 33: Gradient Checking & Debugging, out of order. Put each one back under the part of the lesson it belongs to.

Why check a gradient at all?

A wrong gradient fails silently; The idea: perturb the input, watch the loss

Finite differences, derived

Two ways to straddle the point; The tool: Taylor expansion; Forward difference is only O(ε)

The relative-error test

Why relative, not absolute, error; Reading the number

s1

Why check a gradient at all? is where Lesson 33: Gradient Checking & Debugging puts A wrong gradient fails silently, The idea: perturb the input, watch the loss. Knowing which part of the lesson a problem belongs to is most of knowing which method to reach for.

s2

Finite differences, derived is where Lesson 33: Gradient Checking & Debugging puts Two ways to straddle the point, The tool: Taylor expansion, Forward difference is only O(ε). Knowing which part of the lesson a problem belongs to is most of knowing which method to reach for.

s3

The relative-error test is where Lesson 33: Gradient Checking & Debugging puts Why relative, not absolute, error, Reading the number. Knowing which part of the lesson a problem belongs to is most of knowing which method to reach for.

Step through it

Step through Our running model: logistic regression one row at a time. One of these columns never changes — find it, and say why it cannot.

1. Step 1: sample is 0
2. Step 2: sample is 1
3. Step 3: sample is 2
4. Step 4: sample is 3

Step through it

Step through The loss and the forward pass and watch the numbers move. Now imagine the input ten times bigger: which column is the one that stops this being practical?

1. Step 1: sample is 0
2. Step 2: sample is 1
3. Step 3: sample is 2
4. Step 4: sample is 3

Put in order

Put the moves of Derive the analytic gradient into the order they have to happen.

1. Chain rule: ∂L/∂w = (∂L/∂z)·(∂z/∂w)
2. The cross-entropy + sigmoid combine to ∂L/∂zᵢ = (pᵢ − yᵢ)/N
3. Multiply by ∂z/∂w = Xᵀ

Why: These are the moves of the worked example in the order it makes them, and each one is set up by the one before it. L depends on w only through z = Xw.

Draw it

Translate both ways. First write the expression above as a sentence with no symbols in it at all. Then cover it, and write your sentence back as notation. If the two versions disagree, the disagreement is the thing to fix.

Put in order

Put the moves of Compute the gradient by hand, entry by entry into the order they have to happen.

1. Bias entry = mean(p − y) = (−0.5300 + 0.6593 − 0.5646 + 0.3635)/4
2. x₁ entry = Σ x₁ᵢ(pᵢ−yᵢ)/4 = −0.519076
3. x₂ entry = Σ x₂ᵢ(pᵢ−yᵢ)/4 = 0.021153

Why: These are the moves of the worked example in the order it makes them, and each one is set up by the one before it. Column 0 is all ones, so its dot with (p−y) is just the sum; over N it is the mean.

Annotate

From Compute the gradient by hand, entry by entry — read this one piece at a time. What is each part doing?

On: \( \nabla_w L = \begin{bmatrix} -0.017948 \\ -0.519076 \\ 0.021153 \end{bmatrix} \)

• Column 0 is all ones, so its dot with (p−y) is just the sum; over N it is the mean. = −0.017948.
• Column 1 = [0.5,−1.2,2.0,0.3] dotted with (p−y), divided by 4.
• Column 2 = [−0.3,0.8,0.1,−1.5] dotted with (p−y), divided by 4.

Predict first

The table runs: bias | −0.01794812 | −0.01794812 | ≈2e-11 · x₁ | −0.51907571 | −0.51907571 | ≈1e-11

In Gradient-check it: analytic vs numeric, given the rows so far: what is the next one — the row where coord is x₂?

Correct: x₂ | 0.02115318 | 0.02115318 | 1.88e-10

Why: The relationship between the columns, not the individual numbers, is what generates the next row. Analytic and numeric agree to ~10 digits, far under the 1e-5 threshold.

Fill in the blanks

From Cross-check against torch.autograd — one line has had its right-hand side removed. Put it back.

import numpy as np, torch
torch.manual_seed(0)
X = torch.tensor([[1.,0.5,-0.3],[1.,-1.2,0.8],[1.,2.0,0.1],[1.,0.3,-1.5]])
y = torch.tensor([1.,0.,1.,0.])
w = torch.tensor([0.1,-0.2,0.4], requires_grad=True)
p = torch.sigmoid(X @ w)
L = -torch.mean(ytorch.log(p) + (1-y)torch.log(1-p))
L.backward()
print('torch grad :', w.grad.numpy().round(8))
print('our formula:', (X.T @ (p.detach()-y) / len(y)).numpy().round(8))

Why: p is what everything below it consumes, so the wrong expression here fails later and somewhere else. autograd walks the same chain rule we did by hand and lands on identical numbers (max abs diff 0.0 here — bit-for-bit the same).

Step through it

Step through Cross-check against torch.autograd one row at a time. What is driving the change, and what would the row after the last one be?

1. Step 1: coord is bias
2. Step 2: coord is x₁
3. Step 3: coord is x₂

Fill in the blanks

Bug #1 — a sign flip, with the scaffolding fading: two lines are gone now — fill both.

import numpy as np
f = lambda x: np.sum(x*2np.sin(x))
def numeric(f, x, eps=1e-5):
g = np.zeros_like(x)
for i in range(len(x)):
xp, xm = x.copy(), x.copy(); xp[i]+=eps; xm[i]-=eps
g[i] = (f(xp) - f(xm)) / (2*eps)
return g
x = np.array([0.5,1.0,1.5,2.0])
good = lambda x: 2xnp.sin(x) + x*2np.cos(x)
bad = lambda x: 2xnp.sin(x) - x*2np.cos(x) # sign bug
gn = numeric(f, x)
for name, g in [('good', good(x)), ('bad', bad(x))]:
rel = np.max(np.abs(g-gn)/(np.abs(g)+np.abs(gn)+1e-12))
print(f'___: ___')

Why: Reproducing these unaided, rather than reading them, is what tells you the method has transferred. The correct gradient sits at 1e-11; the sign bug jumps to ~0.46.

Fill in the blanks

From Bug #2 — forgetting the 1/N — one line has had its right-hand side removed. Put it back.

import numpy as np
X = np.array([[1.,0.5,-0.3],[1.,-1.2,0.8],[1.,2.0,0.1],[1.,0.3,-1.5]])
y = np.array([1.,0.,1.,0.]); w = np.array([0.1,-0.2,0.4])
sigmoid = lambda z: 1/(1+np.exp(-z))
def loss(w):
p = sigmoid(X @ w)
return -np.mean(ynp.log(p)+(1-y)np.log(1-p))
def numeric(f, w, eps=1e-5):
g = np.zeros_like(w)
for i in range(len(w)):
wp, wm = w.copy(), w.copy(); wp[i]+=eps; wm[i]-=eps
g[i]=*(f(wp)-f(wm))/(2eps)**
return g
p = sigmoid(X @ w)
gn = numeric(loss, w)
g_bug = X.T @ (p - y) # BUG: no / len(y)
rel = np.max(np.abs(g_bug-gn)/(np.abs(g_bug)+np.abs(gn)+1e-12))
print(f'missing 1/N rel err = ___')

Why: g[i] is what everything below it consumes, so the wrong expression here fails later and somewhere else. Every coordinate is 4× too big.

Put in order

These are the steps of The gradient-check recipe, scrambled. Put them back in order before the next slide shows you.

1. Numeric gradient: for each coordinate i, central-difference with ε = 1e-5: (f(x+εeᵢ) − f(x−εeᵢ)) / 2ε
2. Analytic gradient: the one your backward pass computes
3. Relative error: max |gₐ − gₙ| / (|gₐ| + |gₙ| + 1e-12) over all coordinates
4. Verdict: < 1e-5 correct; ~1e-3 and up is a bug — go fix the formula, not ε
5. Locate: a constant rel error = a scaling bug; a coordinate-specific one = that term's derivative
6. Confirm-correct: for extra assurance, cross-check against torch.autograd

Why: This is the order the recipe itself gives. Recalling the sequence without the slide in front of you is the difference between recognising the method and being able to run it — most of what goes wrong in practice is a step done out of turn.

Eliminate the wrong options

The central difference (f(x+ε)−f(x−ε))/2ε has truncation error of order:

3 of these 4 are wrong. Strike them one at a time, and say what rules each one out before you strike the next. The survivor is the answer.

• A. O(ε²) — the even-power terms cancel, leaving the ε² term
• B. O(ε) — same as the forward difference
• C. O(1/ε) — it grows as ε shrinks
• D. exactly zero — it is the true derivative

Survives elimination: A

Why: Subtracting the two Taylor expansions cancels the constant and ½f''ε² (even-power) terms; the first surviving error term is ⅙f'''ε², so after dividing by 2ε the error is O(ε²). Halving ε quarters the error.

Check your understanding

The central difference (f(x+ε)−f(x−ε))/2ε has truncation error of order:

• A. O(ε²) — the even-power terms cancel, leaving the ε² term (correct)
• B. O(ε) — same as the forward difference
• C. O(1/ε) — it grows as ε shrinks
• D. exactly zero — it is the true derivative

Answer: A

Why: Subtracting the two Taylor expansions cancels the constant and ½f''ε² (even-power) terms; the first surviving error term is ⅙f'''ε², so after dividing by 2ε the error is O(ε²). Halving ε quarters the error.

Why B tempts people

The forward difference is O(ε) because it keeps the ½f''ε term. The central difference is strictly better — that term cancels.

Why C tempts people

O(1/ε) describes the ROUND-OFF error growth as ε → 0, not the truncation error of the formula. The truncation error is O(ε²) and shrinks with ε.

Why D tempts people

It is an approximation: O(ε²) truncation plus round-off. It equals the true derivative only in the ε → 0 limit, which a computer cannot take.

Predict first

Your gradient check returns a max relative error of 0.46. The right move is:

Answer it in your own words, now, with nothing to choose from. The options are on the next slide — and picking the right one off a list is an easier skill than producing it.

Correct: Treat the analytic gradient as buggy and fix the formula

Why: A correct gradient gives rel error around 1e-7 to 1e-11. 0.46 is enormous — the analytic gradient itself is wrong (here, a sign flip). Find and fix the derivation.

Check your understanding

Your gradient check returns a max relative error of 0.46. The right move is:

• A. Treat the analytic gradient as buggy and fix the formula (correct)
• B. Accept it — 0.46 is a small relative error
• C. Shrink ε until the error drops below 1e-5
• D. Ignore it; gradient checks are unreliable noise

Answer: A

Why: A correct gradient gives rel error around 1e-7 to 1e-11. 0.46 is enormous — the analytic gradient itself is wrong (here, a sign flip). Find and fix the derivation.

Why B tempts people

The denominator caps rel error at 1, so 0.46 is nearly the worst possible. Correct gradients are ~1e-9; 0.46 is a definite bug.

Why C tempts people

Shrinking ε changes only the NUMERIC estimate, which is already accurate. It cannot fix a wrong analytic FORMULA — you would just chase round-off.

Why D tempts people

Done right (central diff, ε≈1e-5, relative error), gradient checks are extremely reliable. A 0.46 is real signal, not noise.

Predict first

The single most common silent bug: subtracting a (n,) from a (n,1). NumPy broadcasts them into an (n,n) matrix instead of erroring. Standalone:

Commit before you compute: what does Bug: broadcasting turns a vector into a matrix come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: (3,) − (3,1) → (3,3), not (3,)

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. Broadcasting aligns from the RIGHT: (3,) becomes (1,3), and (1,3) vs (3,1) expands to (3,3).

Sort into buckets

Sort these by shape, from memory, without looking back at Bug: broadcasting turns a vector into a matrix. Telling them apart on the spot is the skill; the table is only where the answer happens to be written down.

(3,)

a (vector); a − b.ravel()

(3, 1)

b (column)

(3, 3)

a − b (BUG)

g1

shape is "(3,)" for a (vector), a − b.ravel() — that is what the table on "Bug: broadcasting turns a vector into a…" records, and it is the single property separating this group from the rest.

g2

shape is "(3, 1)" for b (column) — that is what the table on "Bug: broadcasting turns a vector into a…" records, and it is the single property separating this group from the rest.

g3

shape is "(3, 3)" for a − b (BUG) — that is what the table on "Bug: broadcasting turns a vector into a…" records, and it is the single property separating this group from the rest.

Comparison matrix

From Bug: forgetting zero_grad(): refill the grad (no zero_grad) column from what you know. The rest of the table is as it appeared.

Predict first

A student writes this, and it looks reasonable:

Loss is going UP during training. Obviously the model needs to train longer, or the dataset is too small — crank the epochs and add data.

It is wrong. Say what breaks — and say it before you turn the page.

Correct: A rising TRAINING loss means you are stepping uphill — the update itself is wrong.

A rising training loss = you're doing gradient ascent. Suspect a sign error in the gradient or a learning rate too high.

Why: A rising TRAINING loss means you are stepping uphill — the update itself is wrong. More epochs make it worse, not better; more data doesn't fix a mis-signed gradient or a divergent learning rate.

Discussion prompt

The rule this trap just fixed:

A rising training loss = you're doing gradient ascent. Suspect a sign error in the gradient or a learning rate too high.

Now break it on purpose. Build a case that violates it and follow the consequences until something visibly fails. Where does the failure first show up — and would you have noticed it if you had not been looking?

Hint: The dangerous rules are the ones whose violation still produces an answer. If yours fails loudly, try to find one that fails quietly.

Discussion prompt

Explain Why deep gradients vanish or explode to a student a year behind you. No notation, no jargon they have not met — and it still has to be true.

Hint: If your explanation needs a symbol they have never seen, you are describing the notation rather than the idea.

Fill in the blanks

From Where the 0.25 comes from — one line has had its right-hand side removed. Put it back.

import numpy as np
sig = lambda z: 1/(1+np.exp(-z))
sigp = *lambda z: sig(z)(1-sig(z))** # σ'(z) = σ(1-σ)
for z in [-4, -2, 0, 2, 4]:
print(f'z=___ sigma=___ sigma_prime=___')
print('peak at z=0:', sigp(0))

Why: sigp is what everything below it consumes, so the wrong expression here fails later and somewhere else. At z=0, σ=0.5 and σ'=0.5·0.5=0.25.

Step through it

Step through Where the 0.25 comes from one row at a time. What is driving the change, and what would the row after the last one be?

1. Step 1: z is −4
2. Step 2: z is −2
3. Step 3: z is 0
4. Step 4: z is 2
5. Step 5: z is 4

Predict first

The table runs: 0 | nearest input | 3.74e-02 · 1 | — | 3.37e-01 · 2 | — | 2.67e+00

In Diagnose vanishing with per-layer grad norms, given the rows so far: what is the next one — the row where layer is 3?

Correct: 3 | nearest output | 1.46e+01

Why: The relationship between the columns, not the individual numbers, is what generates the next row. The earliest layer (0) gets a gradient ~400× smaller than the last (3).

Discussion prompt

The cure for exploding gradients is gradient clipping: if the gradient's norm exceeds a threshold, rescale it down to that threshold — same direction, capped length. Standalone:

What do you need to know — or decide — before the first line can be written? List everything the problem has to hand you.

Hint: Anything you would have to invent to get started is a thing the problem must supply.

Predict first

A NaN (or inf) loss almost always comes from log(0), 0/0, sqrt of a negative, or overflow. Cross-entropy hits log(0) when a prediction saturates to exactly 0 or 1. Standalone:

Commit before you compute: what does Hunting NaNs: log(0) and the clip fix come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: Raw loss = inf; clipping p to [1e-12, 1−1e-12] gives 9.4414

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. np.log(0) is −inf, so the mean is inf and every downstream gradient becomes NaN.

Discussion prompt

Work backwards. The example finished here:

Raw loss = inf; clipping p to [1e-12, 1−1e-12] gives 9.4414

What was it asked to do, and what must it have been given? Reconstruct the problem from its answer.

Hint: Every quantity in the result had to enter somewhere. Account for each one.

Predict first

A student writes this, and it looks reasonable:

Write the sigmoid straight from the formula: 1 / (1 + np.exp(-z)). It's the definition — what could go wrong?

It is wrong. Say what breaks — and say it before you turn the page.

Correct: exp(−(−1000)) = exp(1000) = inf → a RuntimeWarning: overflow.

Use the branch-stable form: for z ≥ 0 use 1/(1+e⁻ᶻ); for z < 0 use eᶻ/(1+eᶻ). Each branch only ever exponentiates a non-positive number, so exp stays in [0, 1].

Why: exp(−(−1000)) = exp(1000) = inf → a RuntimeWarning: overflow. The final value happens to round to 0 here, but the inf can propagate (inf−inf = NaN) and the warning is telling you the intermediate blew up. Large-magnitude logits are common once training diverges.

Sort into buckets

Some of these hold up and some are the exact mistakes this lesson is built to prevent. Sort them.

Holds up

Training still runs. The loss still changes. It just heads the wrong way, or the wrong distance, and you waste hours blaming your learning rate or your data.; Everything about finite-difference accuracy falls out of one fact: a smooth f near x is its Taylor series. Expanding at x + ε and at x − ε:; Normalize by the size of the gradients so the verdict is scale-free:

Breaks

Use the forward difference (f(x+ε) − f(x))/ε and make ε as tiny as possible — smaller ε must mean less error, right?; Loss is going UP during training. Obviously the model needs to train longer, or the dataset is too small — crank the epochs and add data.

sound

These are stated as this lesson states them — each one survives the edge cases Lesson 33: Gradient Checking & Debugging puts it through.

flawed

Each of these is lifted from a trap in this deck: reasonable-sounding, and wrong in a way that only shows up once you rely on it.

Discussion prompt

Run The debugging checklist with this step confiscated:

zero_grad: reset gradients each step or they accumulate (2, 4, 6, …)

Is it still possible? If it is, say what takes its place and what it costs you. If it is not, say exactly what that step was providing that nothing else does.

Hint: A step you can drop for free was never load-bearing. If you cannot drop it, name the thing that goes wrong the moment it is gone.

Discussion prompt

The debugging checklist works on the cases you have just seen. Push it to the edge: what is the most degenerate input it still handles — empty, zero, one item, everything equal — and what is the first case where it stops being true? Name the case, not just "it breaks".

Hint: Try the smallest legal input, then the largest, then the one where two things collide. Methods are specified at their edges; the middle takes care of itself.

Predict first

You compute a - b where a.shape is (4,) and b.shape is (4, 1). The result shape is:

Commit to an answer, then rate it — certain, fairly sure, or guessing — and write the rating down before you turn the page.

Correct: (4, 4) — a silent broadcast, almost certainly a bug

Why: Broadcasting aligns from the right: (4,) is treated as (1,4), and (1,4) against (4,1) expands both to (4,4). No error is raised, so a downstream mean() averages 16 values instead of 4 — a classic silent bug. Fix with b.ravel() or matching shapes.

The rating matters as much as the answer: confident-and-wrong is the combination that survives revision, because nothing about it feels like it needs revisiting.

Check your understanding

You compute a - b where a.shape is (4,) and b.shape is (4, 1). The result shape is:

• A. (4, 4) — a silent broadcast, almost certainly a bug (correct)
• B. (4,) — element-wise subtraction
• C. (4, 1) — b's shape wins
• D. a ValueError — the shapes are incompatible

Answer: A

Why: Broadcasting aligns from the right: (4,) is treated as (1,4), and (1,4) against (4,1) expands both to (4,4). No error is raised, so a downstream mean() averages 16 values instead of 4 — a classic silent bug. Fix with b.ravel() or matching shapes.

Why B tempts people

You'd get (4,) only if both operands were (4,) (or one were a scalar). Here b is (4,1), which forces a 2-D broadcast.

Why C tempts people

Broadcasting doesn't pick one operand's shape; it expands each dimension to the max, giving (4,4), not (4,1).

Why D tempts people

These shapes are broadcast-COMPATIBLE — 1 expands to 4 in each direction — so NumPy proceeds silently rather than erroring. That silence is exactly the danger.

Predict first

In PyTorch you call backward() three times on the same constant-gradient step and never call zero_grad(). The .grad values you read are:

Answer it in your own words, now, with nothing to choose from. The options are on the next slide — and picking the right one off a list is an easier skill than producing it.

Correct: 2, 4, 6 — gradients accumulate across backward() calls

Why: PyTorch ADDS each new gradient into .grad rather than overwriting, so a constant gradient of 2 accumulates to 2, 4, 6. You must call optimizer.zero_grad() (or .grad.zero_()) before each backward() to reset it.

Check your understanding

In PyTorch you call backward() three times on the same constant-gradient step and never call zero_grad(). The .grad values you read are:

• A. 2, 4, 6 — gradients accumulate across backward() calls (correct)
• B. 2, 2, 2 — each backward() overwrites the last
• C. 2, 1, 0.67 — they are averaged over calls
• D. a RuntimeError on the second backward()

Answer: A

Why: PyTorch ADDS each new gradient into .grad rather than overwriting, so a constant gradient of 2 accumulates to 2, 4, 6. You must call optimizer.zero_grad() (or .grad.zero_()) before each backward() to reset it.

Why B tempts people

Overwriting is what you WANT, but it's not the default — you get it only by calling zero_grad() first. Without it, .grad accumulates.

Why C tempts people

PyTorch never averages gradients across backward() calls; it sums them. Averaging over a batch happens inside a single loss (e.g. mean), not across steps.

Why D tempts people

A second backward() errors only if you didn't retain the graph AND are reusing the same graph. Here each iteration builds a fresh graph, so it accumulates without error.

Eliminate the wrong options

A binary-cross-entropy loss turns NaN after a few steps. The most likely immediate cause is:

3 of these 4 are wrong. Strike them one at a time, and say what rules each one out before you strike the next. The survivor is the answer.

• A. A prediction saturated to exactly 0 or 1, so log(0) = −inf
• B. The learning rate is slightly too low
• C. The batch size is not a power of two
• D. You used float64 instead of float32

Survives elimination: A

Why: BCE contains log(p) and log(1−p). When a prediction saturates to 0 or 1, one of those logs is log(0) = −inf, which propagates to a NaN loss and NaN gradients. Clamp p to [1e-12, 1−1e-12] (or use a log-sum-exp-stable loss).

Check your understanding

A binary-cross-entropy loss turns NaN after a few steps. The most likely immediate cause is:

• A. A prediction saturated to exactly 0 or 1, so log(0) = −inf (correct)
• B. The learning rate is slightly too low
• C. The batch size is not a power of two
• D. You used float64 instead of float32

Answer: A

Why: BCE contains log(p) and log(1−p). When a prediction saturates to 0 or 1, one of those logs is log(0) = −inf, which propagates to a NaN loss and NaN gradients. Clamp p to [1e-12, 1−1e-12] (or use a log-sum-exp-stable loss).

Why B tempts people

A too-low learning rate makes training SLOW, not NaN. NaNs come from invalid math (log(0), 0/0, overflow), not from small steps.

Why C tempts people

Batch size being a power of two is a minor performance preference; it has no effect on whether the loss is finite.

Why D tempts people

float64 is MORE precise than float32 and less prone to overflow, so it would reduce NaNs, not cause them. The culprit is log(0), independent of dtype.

Discussion prompt

Explain Project: your reusable gradient checker by analogy to something with no Machine Learning in it at all — a queue, a recipe, a map, a bank balance, whatever fits. Then say where your analogy breaks.

Hint: An analogy that never breaks is not an analogy, it is the same idea wearing a hat. Find the seam — that is the part that is actually new.

Step through it

Step through Milestone 1 — the numeric gradient and watch the numbers move. Now imagine the input ten times bigger: which column is the one that stops this being practical?

1. Step 1: x is 0.5
2. Step 2: x is 1.0
3. Step 3: x is 1.5
4. Step 4: x is 2.0

Step through it

Step through Milestone 2 — check the correct gradient one row at a time. What is driving the change, and what would the row after the last one be?

1. Step 1: quantity is max rel error
2. Step 2: quantity is threshold
3. Step 3: quantity is pass?

Comparison matrix

From Milestone 4 — check a real model gradient: every row here is a choice with a cost. Fill the rel error column, then say which row you would actually pick and what you give up for it.

Comparison matrix

From The full program: refill the rel error column from what you know. The rest of the table is as it appeared.

Discussion prompt

Slides closed, out loud: explain (1) why the central difference is O(ε²) — which Taylor terms cancel — (2) why ε = 1e-12 is worse than 1e-5, and (3) what a rising training loss tells you.

That is stated as though it always holds. Do one of two things: produce a case where it fails, or say precisely what rules such a case out. "It just does" is not on the menu.

Hint: Hunt at the extremes first — zero, one, negative, empty, equal. If every extreme survives, the reason they survive is the proof.

Draw it

One page, no notation unless you need it: draw how these connect — Why check a gradient at all? · Finite differences, derived · The relative-error test · Check a real gradient · Now break it · Debugging real models. Put an arrow wherever one of them is what makes another possible, and label the arrow with why.