Lesson 74: Neural Network Debugging Checklist

Discussion prompt

Before we open Lesson 74: Neural Network Debugging Checklist: without looking back, what was the main idea of Metric Learning, Siamese Networks & Triplet Loss, and what could you do by the end of it that you could not do before?

Hint: One sentence for the idea, one for the skill. If the second one is blank, that is the part to revisit.

Discussion prompt

Before tuning anything, verify your model can learn. A healthy network should drive training loss to ~0 on a batch of 8–32 samples within a few hundred steps.

That is stated as though it always holds. Do one of two things: produce a case where it fails, or say precisely what rules such a case out. "It just does" is not on the menu.

Hint: Hunt at the extremes first — zero, one, negative, empty, equal. If every extreme survives, the reason they survive is the proof.

Predict first

4-class network, batch of 8. Predict: will loss reach ~0?

Commit before you compute: what does Overfit a batch of 8 — loss trace come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: Loss falls from 1.41 to ~0 by epoch 20; batch accuracy hits 1.0 by epoch 5

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. The network can learn the data — no structural bug.

Comparison matrix

From Overfit a batch of 8 — loss trace: refill the loss column from what you know. The rest of the table is as it appeared.

Predict first

A student writes this, and it looks reasonable:

Training loss is stuck at 1.5 after 1000 steps. The first move: lower the learning rate.

It is wrong. Say what breaks — and say it before you turn the page.

Correct: A stuck loss may have nothing to do with the learning rate.

Training loss is stuck at 1.5. First: overfit a single batch.

Why: A stuck loss may have nothing to do with the learning rate. Lowering it on a broken model wastes GPU hours and produces no signal.

Discussion prompt

The rule this trap just fixed:

If not, the bug is structural (wrong loss, bad init, missing activation). Fix the bug first. Tuning hyperparameters is Step 5, not Step 1.

Now break it on purpose. Build a case that violates it and follow the consequences until something visibly fails. Where does the failure first show up — and would you have noticed it if you had not been looking?

Hint: The dangerous rules are the ones whose violation still produces an answer. If yours fails loudly, try to find one that fails quietly.

Comparison matrix

From Gradient norms reveal where learning stops: every row here is a choice with a cost. Fill the grad norm pattern column, then say which row you would actually pick and what you give up for it.

Discussion prompt

Two architectures: 2-layer ReLU (healthy) vs 3-layer Sigmoid (vanishing). Same data, one backward pass.

What do you need to know — or decide — before the first line can be written? List everything the problem has to hand you.

Hint: Anything you would have to invent to get started is a thing the problem must supply.

Discussion prompt

Work backwards. The example finished here:

Sigmoid early layers: norms ~0.007–0.011, 50× smaller than the final layer; ReLU layers: norms stay in the same order throughout

What was it asked to do, and what must it have been given? Reconstruct the problem from its answer.

Hint: Every quantity in the result had to enter somewhere. Account for each one.

Step through it

Step through Random-classifier baseline: loss = log(K) one row at a time. What is driving the change, and what would the row after the last one be?

1. Step 1: K (classes) is 2
2. Step 2: K (classes) is 4
3. Step 3: K (classes) is 5
4. Step 4: K (classes) is 10
5. Step 5: K (classes) is 100

Predict first

K=10, 100 samples. Compare near-zero init (standard) vs huge init (σ=10.0).

Commit before you compute: what does Verifying initial loss — near-zero init vs bad init come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: Standard init → loss 2.298 ≈ log(10) = 2.303; huge init → loss 5051 (>1000× too high)

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. Large random weights produce extreme logits; softmax saturates; CE can be enormous.

Discussion prompt

Work backwards. The example finished here:

Standard init → loss 2.298 ≈ log(10) = 2.303; huge init → loss 5051 (>1000× too high)

What was it asked to do, and what must it have been given? Reconstruct the problem from its answer.

Hint: Every quantity in the result had to enter somewhere. Account for each one.

Step through it

Step through Initial accuracy baseline: 1/K one row at a time. What is driving the change, and what would the row after the last one be?

1. Step 1: K is 2 (binary)
2. Step 2: K is 4
3. Step 3: K is 10
4. Step 4: K is 100

Predict first

A student writes this, and it looks reasonable:

4-class classification. Use MSELoss(logits, y) where y contains class indices 0–3 — training loss goes down, so it must be learning.

It is wrong. Say what breaks — and say it before you turn the page.

Correct: MSE minimizes numeric distance to class indices, not probability calibration.

For K-class classification: use CrossEntropyLoss(logits, class_indices).

Why: MSE minimizes numeric distance to class indices, not probability calibration. The gradients point in a meaningless direction. Loss falling is not evidence of correct learning.

Sort into buckets

Some of these hold up and some are the exact mistakes this lesson is built to prevent. Sort them.

Holds up

Before tuning anything, verify your model can learn. A healthy network should drive training loss to ~0 on a batch of 8–32 samples within a few hundred steps.; After one backward pass, compute param.grad.norm() for every named layer. A healthy network shows similar-order norms throughout the stack.; Hyperparameter tuning is the last step, not the first. Use proven defaults from the literature while debugging — they eliminate hyperparameters as a confound.

Breaks

Training loss is stuck at 1.5 after 1000 steps. The first move: lower the learning rate.; 4-class classification. Use MSELoss(logits, y) where y contains class indices 0–3 — training loss goes down, so it must be learning.

sound

These are stated as this lesson states them — each one survives the edge cases Lesson 74: Neural Network Debugging Checklist puts it through.

flawed

Each of these is lifted from a trap in this deck: reasonable-sounding, and wrong in a way that only shows up once you rely on it.

Comparison matrix

From Start from literature defaults, not a random search: refill the reliable default column from what you know. The rest of the table is as it appeared.

Discussion prompt

Encode all five checks as a single Python function. Run it before every new experiment.

What do you need to know — or decide — before the first line can be written? List everything the problem has to hand you.

Hint: Anything you would have to invent to get started is a thing the problem must supply.

Sort into buckets

Sort these by healthy?, from memory, without looking back at The automated debugging checklist function. Telling them apart on the spot is the skill; the table is only where the answer happens to be written down.

True

init_loss vs log(K=4); can_overfit (ep=100)

True (within 0.15)

init_acc vs 1/4

False (no vanishing)

vanishing grads

g1

healthy? is "True" for init_loss vs log(K=4), can_overfit (ep=100) — that is what the table on "The automated debugging checklist…" records, and it is the single property separating this group from the rest.

g2

healthy? is "True (within 0.15)" for init_acc vs 1/4 — that is what the table on "The automated debugging checklist…" records, and it is the single property separating this group from the rest.

g3

healthy? is "False (no vanishing)" for vanishing grads — that is what the table on "The automated debugging checklist…" records, and it is the single property separating this group from the rest.

Put in order

These are the steps of The 5-step debugging protocol, scrambled. Put them back in order before the next slide shows you.

1. Overfit a single batch — if loss can't reach ~0, fix architecture before anything else
2. Check gradient norms per layer — near-zero early norms = vanishing; astronomic = exploding
3. Check initial loss against log(K) — far above log(K) = bad init or missing normalization
4. Check initial accuracy against 1/K — inconsistent pair = shape bug or label leak
5. Use literature defaults first, then tune — hyperparameter search on a broken model yields noise

Why: This is the order the recipe itself gives. Recalling the sequence without the slide in front of you is the difference between recognising the method and being able to run it — most of what goes wrong in practice is a step done out of turn.

Discussion prompt

The 5-step debugging protocol works on the cases you have just seen. Push it to the edge: what is the most degenerate input it still handles — empty, zero, one item, everything equal — and what is the first case where it stops being true? Name the case, not just "it breaks".

Hint: Try the smallest legal input, then the largest, then the one where two things collide. Methods are specified at their edges; the middle takes care of itself.

Eliminate the wrong options

Your 10-class classifier shows initial cross-entropy loss of 4.8. What does this most likely indicate?

3 of these 4 are wrong. Strike them one at a time, and say what rules each one out before you strike the next. The survivor is the answer.

• A. Bad weight initialization — logits are far from uniform, pushing loss well above log(10) = 2.30
• B. The model is already learning fast — high initial loss means strong early gradients
• C. Correct behavior — cross-entropy for 10 classes is always around 4–5
• D. The learning rate is too high, causing the optimizer to overshoot before the first evaluation

Survives elimination: A

Why: A randomly initialized network with near-zero weights produces uniform logits → uniform softmax → CE ≈ log(10) = 2.303. A loss of 4.8 (≈ log(121)) means logits are highly non-uniform at init, which points to large random weights (bad initialization) or a forgotten input normalization.

Check your understanding

Your 10-class classifier shows initial cross-entropy loss of 4.8. What does this most likely indicate?

• A. Bad weight initialization — logits are far from uniform, pushing loss well above log(10) = 2.30 (correct)
• B. The model is already learning fast — high initial loss means strong early gradients
• C. Correct behavior — cross-entropy for 10 classes is always around 4–5
• D. The learning rate is too high, causing the optimizer to overshoot before the first evaluation

Answer: A

Why: A randomly initialized network with near-zero weights produces uniform logits → uniform softmax → CE ≈ log(10) = 2.303. A loss of 4.8 (≈ log(121)) means logits are highly non-uniform at init, which points to large random weights (bad initialization) or a forgotten input normalization.

Why B tempts people

High initial loss does produce large gradients, but that is a consequence of the bug, not a sign of healthy learning. The network hasn't taken a step yet — there is no 'learning fast'.

Why C tempts people

The correct baseline for K=10 is log(10) = 2.303, not 4–5. An answer of 4.8 is more than 2× above the expected value and is a clear red flag.

Why D tempts people

Initial loss is measured before any optimizer step, so the learning rate cannot have affected it yet.

Predict first

After one backward pass, grad norms are: layer1=0.007, layer2=0.009, layer3=0.058, layer4=0.37. What is the most likely cause and fix?

Answer it in your own words, now, with nothing to choose from. The options are on the next slide — and picking the right one off a list is an easier skill than producing it.

Correct: Vanishing gradients — early layers barely update; switch to ReLU or add batch normalization

Why: Norms of 0.007–0.009 in early layers vs 0.37 in the last layer indicate vanishing gradients — each Sigmoid gate multiplies the backward signal by σ'≤0.25, shrinking it ~50× across three layers. Early layers barely learn. Fix: replace Sigmoid with ReLU, or add BatchNorm to keep activations in the linear regime.

Check your understanding

After one backward pass, grad norms are: layer1=0.007, layer2=0.009, layer3=0.058, layer4=0.37. What is the most likely cause and fix?

• A. Vanishing gradients — early layers barely update; switch to ReLU or add batch normalization (correct)
• B. Exploding gradients — norms are too large; add gradient clipping
• C. Normal behavior — gradients are always smaller in earlier layers
• D. The learning rate is too low — raise lr to increase gradient magnitude

Answer: A

Why: Norms of 0.007–0.009 in early layers vs 0.37 in the last layer indicate vanishing gradients — each Sigmoid gate multiplies the backward signal by σ'≤0.25, shrinking it ~50× across three layers. Early layers barely learn. Fix: replace Sigmoid with ReLU, or add BatchNorm to keep activations in the linear regime.

Why B tempts people

Exploding gradients would show norms >> 1 (often 1e3–1e6 and NaN loss). Values of 0.007–0.37 are uniformly small — the problem is vanishing, not exploding.

Why C tempts people

Gradients being slightly smaller in earlier layers is normal due to the chain rule, but a 50× ratio (0.007 vs 0.37) is pathological, not typical. A healthy ReLU net shows same-order norms throughout.

Why D tempts people

Gradient magnitude is set by the loss landscape and architecture, not by learning rate. Raising lr scales the update step but doesn't change the gradient norm computed by backward().

Eliminate the wrong options

A network's training loss decreases smoothly but validation accuracy stays at chance (1/K) throughout training. Which bug is most consistent with this pattern?

3 of these 4 are wrong. Strike them one at a time, and say what rules each one out before you strike the next. The survivor is the answer.

• A. Label leak in the training set — the model memorizes train labels but the validation labels are correct and unseen
• B. Vanishing gradients — early layers don't update, so the network can't generalize
• C. Missing ReLU activation — without nonlinearity the network has no expressive power
• D. Wrong loss function (MSE instead of CE) — the model trains on a corrupted signal

Survives elimination: A

Why: Decreasing training loss with validation accuracy stuck at 1/K is the signature of a train/validation data pipeline bug. Most commonly: train and validation sets share samples (so the model memorizes train labels), or train labels were accidentally permuted before splitting while validation retains the real labels. The model learns something on train but that information doesn't transfer to clean validation data.

Check your understanding

A network's training loss decreases smoothly but validation accuracy stays at chance (1/K) throughout training. Which bug is most consistent with this pattern?

• A. Label leak in the training set — the model memorizes train labels but the validation labels are correct and unseen (correct)
• B. Vanishing gradients — early layers don't update, so the network can't generalize
• C. Missing ReLU activation — without nonlinearity the network has no expressive power
• D. Wrong loss function (MSE instead of CE) — the model trains on a corrupted signal

Answer: A

Why: Decreasing training loss with validation accuracy stuck at 1/K is the signature of a train/validation data pipeline bug. Most commonly: train and validation sets share samples (so the model memorizes train labels), or train labels were accidentally permuted before splitting while validation retains the real labels. The model learns something on train but that information doesn't transfer to clean validation data.

Why B tempts people

Vanishing gradients would cause training loss to also plateau (early layers don't update → limited model capacity → training accuracy also stagnates). A smoothly decreasing training loss rules out severe gradient problems.

Why C tempts people

A linear-only network (no activation) still has representational power for linearly separable data. It would typically show both training and validation loss decreasing, just more slowly — not validation stuck at 1/K.

Why D tempts people

MSE on class indices still provides a gradient signal that often correlates with the correct class ordering. Both train and validation accuracy would likely show some above-chance improvement, not validation pinned at 1/K.

Discussion prompt

Explain Project: build and stress-test the debugging checklist by analogy to something with no Machine Learning in it at all — a queue, a recipe, a map, a bank balance, whatever fits. Then say where your analogy breaks.

Hint: An analogy that never breaks is not an analogy, it is the same idea wearing a hat. Find the seam — that is the part that is actually new.

Comparison matrix

From Milestone 2 — inject and catch bad initialization: every row here is a choice with a cost. Fill the healthy net column, then say which row you would actually pick and what you give up for it.

Predict first

Your turn: build a 3-layer Sigmoid network. Run debug_checklist. Which flags fire?

Commit before you compute: what does Milestone 3 — inject and catch vanishing gradients come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: Early layers: grad_norm < 0.01, 50× below last layer; vanishing=True flag fires

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. Fix: replace nn.Sigmoid() with nn.ReLU() throughout the hidden stack.

Comparison matrix

From Milestone 3 — inject and catch vanishing gradients: refill the grad norm column from what you know. The rest of the table is as it appeared.

Sort into buckets

Sort these by value, from memory, without looking back at The full debugging program. Telling them apart on the spot is the skill; the table is only where the answer happens to be written down.

1.397 / 1.386

init_loss / expected_loss

True

loss_ok; can_overfit

False

vanishing

g1

value is "1.397 / 1.386" for init_loss / expected_loss — that is what the table on "The full debugging program" records, and it is the single property separating this group from the rest.

g2

value is "True" for loss_ok, can_overfit — that is what the table on "The full debugging program" records, and it is the single property separating this group from the rest.

g3

value is "False" for vanishing — that is what the table on "The full debugging program" records, and it is the single property separating this group from the rest.

Discussion prompt

Out loud, slides closed: walk through the five steps in order and explain what each one catches. Then inject one bug of your choice, run the checklist, and narrate which flag fires and why.

That is stated as though it always holds. Do one of two things: produce a case where it fails, or say precisely what rules such a case out. "It just does" is not on the menu.

Hint: Hunt at the extremes first — zero, one, negative, empty, equal. If every extreme survives, the reason they survive is the proof.

Draw it

One page, no notation unless you need it: draw how these connect — Step 1: Overfit a single batch · Step 2: Gradient flow · Step 3: Check loss scale · Step 4: Check metrics · Step 5: Known hyperparameters first · Your turn: debug it. Put an arrow wherever one of them is what makes another possible, and label the arrow with why.