Lesson 77: Deep Learning Foundations

Discussion prompt

Before we open Lesson 77: Deep Learning Foundations: without looking back, what was the main idea of Algorithm Review & Selection, and what could you do by the end of it that you could not do before?

Hint: One sentence for the idea, one for the skill. If the second one is blank, that is the part to revisit.

Sort into buckets

Sort these by operation, from memory, without looking back at MLP: the baseline deep architecture. Telling them apart on the spot is the skill; the table is only where the answer happens to be written down.

—

input

xW^T + b

Linear(64→128); Linear(128→10)

max(0, z)

ReLU

g1

operation is "—" for input — that is what the table on "MLP: the baseline deep architecture" records, and it is the single property separating this group from the rest.

g2

operation is "xW^T + b" for Linear(64→128), Linear(128→10) — that is what the table on "MLP: the baseline deep architecture" records, and it is the single property separating this group from the rest.

g3

operation is "max(0, z)" for ReLU — that is what the table on "MLP: the baseline deep architecture" records, and it is the single property separating this group from the rest.

Comparison matrix

From CNN: exploit spatial structure: refill the params column from what you know. The rest of the table is as it appeared.

Discussion prompt

A residual block outputs relu(F(x) + x) where x is the skip. The gradient of the sum is dF/dx + 1 — the +1 ensures a gradient highway past the block.

That is stated as though it always holds. Do one of two things: produce a case where it fails, or say precisely what rules such a case out. "It just does" is not on the menu.

Hint: Hunt at the extremes first — zero, one, negative, empty, equal. If every extreme survives, the reason they survive is the proof.

Comparison matrix

From Activation functions: output & gradient ranges: every row here is a choice with a cost. Fill the range column, then say which row you would actually pick and what you give up for it.

Discussion prompt

Explain Vanishing gradient: the math by analogy to something with no Machine Learning in it at all — a queue, a recipe, a map, a bank balance, whatever fits. Then say where your analogy breaks.

Hint: An analogy that never breaks is not an analogy, it is the same idea wearing a hat. Find the seam — that is the part that is actually new.

Predict first

A student writes this, and it looks reasonable:

Sigmoid is a 'proper' probability output, so it should work well everywhere — use it for all hidden layers in a 10-layer network.

It is wrong. Say what breaks — and say it before you turn the page.

Correct: Each sigmoid multiplies the backward gradient by ≤0.25.

Use ReLU (or LeakyReLU / GELU) in hidden layers; reserve sigmoid for the output of a binary classifier.

Why: Each sigmoid multiplies the backward gradient by ≤0.25. After 10 layers: 0.25^10 ≈ 9.5×10⁻⁷. Early layers receive essentially zero gradient and their weights stop learning.

Discussion prompt

The rule this trap just fixed:

ReLU's derivative is 1 for positive activations, so the gradient can flow through arbitrarily many layers without shrinking. Sigmoid/tanh belong at the output, not inside the stack.

Now break it on purpose. Build a case that violates it and follow the consequences until something visibly fails. Where does the failure first show up — and would you have noticed it if you had not been looking?

Hint: The dangerous rules are the ones whose violation still produces an answer. If yours fails loudly, try to find one that fails quietly.

Predict first

BatchNorm standardizes a mini-batch of pre-activations to mean=0, std=1, then rescales with learnable γ, β. Input batch: [1, 3, 5, 7].

Commit before you compute: what does Batch normalization — step-by-step come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: μ = 4.0, σ = 2.2361; x̂ values verified from execution

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. BatchNorm eliminates internal covariate shift: each layer always sees inputs near (0,1), stabilizing training and allowing higher learning rates.

Step through it

Step through Batch normalization — step-by-step one row at a time. What is driving the change, and what would the row after the last one be?

1. Step 1: x is 1.0
2. Step 2: x is 3.0
3. Step 3: x is 5.0
4. Step 4: x is 7.0

Discussion prompt

Explain Dropout: inverted scaling to a student a year behind you. No notation, no jargon they have not met — and it still has to be true.

Hint: If your explanation needs a symbol they have never seen, you are describing the notation rather than the idea.

Sort into buckets

These are the pieces of Lesson 77: Deep Learning Foundations, out of order. Put each one back under the part of the lesson it belongs to.

Architecture: MLP → CNN → ResNet

MLP: the baseline deep architecture; CNN: exploit spatial structure; ResNet: skip connections defeat vanishing gradients

Activations & Vanishing Gradients

Activation functions: output & gradient ranges; Vanishing gradient: the math

BatchNorm, Dropout & Initialization

Batch normalization — step-by-step; Dropout: inverted scaling; Weight initialization: Xavier vs Kaiming

s1

Architecture: MLP → CNN → ResNet is where Lesson 77: Deep Learning Foundations puts MLP: the baseline deep architecture, CNN: exploit spatial structure, ResNet: skip connections defeat vanishing gradients. Knowing which part of the lesson a problem belongs to is most of knowing which method to reach for.

s2

Activations & Vanishing Gradients is where Lesson 77: Deep Learning Foundations puts Activation functions: output & gradient ranges, Vanishing gradient: the math. Knowing which part of the lesson a problem belongs to is most of knowing which method to reach for.

s3

BatchNorm, Dropout & Initialization is where Lesson 77: Deep Learning Foundations puts Batch normalization — step-by-step, Dropout: inverted scaling, Weight initialization: Xavier vs Kaiming. Knowing which part of the lesson a problem belongs to is most of knowing which method to reach for.

Predict first

2-layer network. x=[1,2], w1=[[0.5,-0.3],[0.2,0.8]], w2=[[0.4,-0.1]]. Loss = z2.sum(). Trace forward then backward.

Commit before you compute: what does Backpropagation: chain-rule trace come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: Row 0 of dL/dw1 is [0,0] because z1[0]=-0.1 < 0: ReLU killed it, its gradient is 0

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. Chain rule: dL/dw1[i,j] = dL/dz2 · w2[i] · relu'(z1[i]) · x[j].

Comparison matrix

From Backpropagation: chain-rule trace: refill the note column from what you know. The rest of the table is as it appeared.

Sort into buckets

Sort these by extra cost, from memory, without looking back at Optimizer review: SGD → Adam → AdamW. Telling them apart on the spot is the skill; the table is only where the answer happens to be written down.

none

SGD

+v buffer

Momentum

+m,v buffers

Adam; AdamW

g1

extra cost is "none" for SGD — that is what the table on "Optimizer review: SGD → Adam → AdamW" records, and it is the single property separating this group from the rest.

g2

extra cost is "+v buffer" for Momentum — that is what the table on "Optimizer review: SGD → Adam → AdamW" records, and it is the single property separating this group from the rest.

g3

extra cost is "+m,v buffers" for Adam, AdamW — that is what the table on "Optimizer review: SGD → Adam → AdamW" records, and it is the single property separating this group from the rest.

Discussion prompt

Explain Learning rate scheduling by analogy to something with no Machine Learning in it at all — a queue, a recipe, a map, a bank balance, whatever fits. Then say where your analogy breaks.

Hint: An analogy that never breaks is not an analogy, it is the same idea wearing a hat. Find the seam — that is the part that is actually new.

Predict first

A student writes this, and it looks reasonable:

Each iteration: sched.step() first to set the rate, then opt.step() to update weights with that rate.

It is wrong. Say what breaks — and say it before you turn the page.

Correct: PyTorch's LR schedulers expect to be called AFTER the optimizer step.

Call sched.step() at the end of each epoch, after opt.step().

Why: PyTorch's LR schedulers expect to be called AFTER the optimizer step. Calling sched first shifts the schedule by one epoch, meaning epoch 1 uses the epoch-0 rate, epoch 2 uses epoch-1, etc. — subtle and silent.

Sort into buckets

Some of these hold up and some are the exact mistakes this lesson is built to prevent. Sort them.

Holds up

A multi-layer perceptron (Lesson 40) stacks Linear → activation blocks. Every unit sees every input — no spatial structure is exploited.; A Conv2d(1, 8, kernel_size=3, padding=1) on a 1×8×8 input slides an 8 × (1×3×3) filter bank across the image, outputting 8×8×8 feature maps.; A residual block outputs relu(F(x) + x) where x is the skip. The gradient of the sum is dF/dx + 1 — the +1 ensures a gradient highway past the block.

Breaks

Sigmoid is a 'proper' probability output, so it should work well everywhere — use it for all hidden layers in a 10-layer network.; Each iteration: sched.step() first to set the rate, then opt.step() to update weights with that rate.

sound

These are stated as this lesson states them — each one survives the edge cases Lesson 77: Deep Learning Foundations puts it through.

flawed

Each of these is lifted from a trap in this deck: reasonable-sounding, and wrong in a way that only shows up once you rely on it.

Predict first

Wrap raw data in a Dataset subclass, then a DataLoader handles batching, shuffling, and (optionally) multi-process prefetch.

Commit before you compute: what does Dataset + DataLoader come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: 1797 samples ÷ batch 64 = 28 full batches + 1 batch of 5

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. DataLoader automatically handles the partial last batch; set drop_last=True if your model requires a fixed batch size (e.g.

Discussion prompt

Work backwards. The example finished here:

1797 samples ÷ batch 64 = 28 full batches + 1 batch of 5

What was it asked to do, and what must it have been given? Reconstruct the problem from its answer.

Hint: Every quantity in the result had to enter somewhere. Account for each one.

Discussion prompt

Run The complete deep-learning pipeline with this step confiscated:

Regularize: BatchNorm (stable activations) + Dropout (generalization)

Is it still possible? If it is, say what takes its place and what it costs you. If it is not, say exactly what that step was providing that nothing else does.

Hint: A step you can drop for free was never load-bearing. If you cannot drop it, name the thing that goes wrong the moment it is gone.

Discussion prompt

The complete deep-learning pipeline works on the cases you have just seen. Push it to the edge: what is the most degenerate input it still handles — empty, zero, one item, everything equal — and what is the first case where it stops being true? Name the case, not just "it breaks".

Hint: Try the smallest legal input, then the largest, then the one where two things collide. Methods are specified at their edges; the middle takes care of itself.

Eliminate the wrong options

A 16-layer network trained with sigmoid hidden activations shows near-zero weight updates in the first few layers. Which change MOST directly fixes this?

3 of these 4 are wrong. Strike them one at a time, and say what rules each one out before you strike the next. The survivor is the answer.

• A. Replace sigmoid hidden activations with ReLU
• B. Increase the learning rate by 10×
• C. Add more hidden units to each layer
• D. Switch from Adam to SGD

Survives elimination: A

Why: Sigmoid's max derivative is 0.25, so 16 layers multiply the gradient by ≤0.25^16 ≈ 2.3×10⁻¹⁰. ReLU has derivative 1 for positive activations, preventing this multiplicative decay. A higher lr or more units doesn't fix the root cause.

Check your understanding

A 16-layer network trained with sigmoid hidden activations shows near-zero weight updates in the first few layers. Which change MOST directly fixes this?

• A. Replace sigmoid hidden activations with ReLU (correct)
• B. Increase the learning rate by 10×
• C. Add more hidden units to each layer
• D. Switch from Adam to SGD

Answer: A

Why: Sigmoid's max derivative is 0.25, so 16 layers multiply the gradient by ≤0.25^16 ≈ 2.3×10⁻¹⁰. ReLU has derivative 1 for positive activations, preventing this multiplicative decay. A higher lr or more units doesn't fix the root cause.

Why B tempts people

A higher lr scales the already-near-zero gradient — it multiplies zero, not un-vanishes it. It can also cause divergence in the later layers where gradients are healthy.

Why C tempts people

More units increase capacity but don't change the per-unit gradient magnitude. The vanishing problem is about the chain-rule multiplier, not the number of neurons.

Why D tempts people

The choice of optimizer (Adam vs SGD) does not fix vanishing gradients — both multiply the same near-zero gradient by a step-size. The problem is in the activation function, not the optimizer.

Predict first

During training, BatchNorm normalizes each mini-batch using the batch's mean and variance. At inference (model.eval()), what does BatchNorm use instead?

Answer it in your own words, now, with nothing to choose from. The options are on the next slide — and picking the right one off a list is an easier skill than producing it.

Correct: Running (exponential moving) mean and variance accumulated during training

Why: PyTorch's BatchNorm tracks running_mean and running_var via momentum during training. At eval() it switches to these fixed statistics so the output is deterministic for a single sample — you don't need a batch.

Check your understanding

During training, BatchNorm normalizes each mini-batch using the batch's mean and variance. At inference (model.eval()), what does BatchNorm use instead?

• A. Running (exponential moving) mean and variance accumulated during training (correct)
• B. Mean and variance of the current test batch
• C. All zeros for mean and all ones for variance (identity)
• D. The learnable parameters γ and β directly, with no normalization

Answer: A

Why: PyTorch's BatchNorm tracks running_mean and running_var via momentum during training. At eval() it switches to these fixed statistics so the output is deterministic for a single sample — you don't need a batch.

Why B tempts people

Using test-batch statistics would make inference non-deterministic for single samples (batch_size=1 gives var=0) and would require knowing the test distribution — exactly what we don't want.

Why C tempts people

Passing through unnormalized (identity) would undo all the benefits of BatchNorm and produce a distribution mismatch between training and inference.

Why D tempts people

γ and β are the post-normalization scale and shift; they're applied after the standardization step, not instead of it. Skipping normalization entirely breaks the layer.

Eliminate the wrong options

After calling loss.backward(), the next two calls in the PyTorch training loop should be, in order:

3 of these 4 are wrong. Strike them one at a time, and say what rules each one out before you strike the next. The survivor is the answer.

• A. opt.step(), then opt.zero_grad() for the NEXT iteration
• B. opt.zero_grad(), then opt.step()
• C. opt.step(), then loss.backward() again for momentum
• D. sched.step(), then opt.step()

Survives elimination: A

Why: backward() fills the .grad tensors; opt.step() reads those grads and updates parameters; then zero_grad() clears them so the NEXT iteration starts clean. sched.step() goes at the end of the epoch, not each batch.

Check your understanding

After calling loss.backward(), the next two calls in the PyTorch training loop should be, in order:

• A. opt.step(), then opt.zero_grad() for the NEXT iteration (correct)
• B. opt.zero_grad(), then opt.step()
• C. opt.step(), then loss.backward() again for momentum
• D. sched.step(), then opt.step()

Answer: A

Why: backward() fills the .grad tensors; opt.step() reads those grads and updates parameters; then zero_grad() clears them so the NEXT iteration starts clean. sched.step() goes at the end of the epoch, not each batch.

Why B tempts people

zero_grad() before opt.step() would wipe the gradients backward() just computed — the optimizer would step with all-zero gradients and parameters would not update.

Why C tempts people

Calling backward() twice without zero_grad() accumulates gradients, doubling the effective gradient. There is no 'momentum backward' — momentum lives entirely inside the optimizer.

Why D tempts people

sched.step() updates the learning rate for the next epoch, not the next batch. Calling it before opt.step() shifts the schedule by one epoch and applies the wrong rate to the current step.

Discussion prompt

Build a complete digits classifier applying every technique from today: BatchNorm, Dropout, Kaiming init, Adam optimizer, StepLR schedule, and a DataLoader mini-batch loop.

That is stated as though it always holds. Do one of two things: produce a case where it fails, or say precisely what rules such a case out. "It just does" is not on the menu.

Hint: Hunt at the extremes first — zero, one, negative, empty, equal. If every extreme survives, the reason they survive is the proof.

Comparison matrix

From Milestone 2 — MLP + training loop: every row here is a choice with a cost. Fill the expected loss column, then say which row you would actually pick and what you give up for it.

Comparison matrix

From The full program: refill the role column from what you know. The rest of the table is as it appeared.

Draw it

One page, no notation unless you need it: draw how these connect — Architecture: MLP → CNN → ResNet · Activations & Vanishing Gradients · BatchNorm, Dropout & Initialization · Backprop & Optimizers · Data Pipeline & Pattern · Your turn: full digits classifier. Put an arrow wherever one of them is what makes another possible, and label the arrow with why.