Lesson 44: MLP Architectures — Depth, Width & Activations

Discussion prompt

Before we open Lesson 44: MLP Architectures — Depth, Width & Activations: without looking back, what was the main idea of Support Vector Machines, and what could you do by the end of it that you could not do before?

Hint: One sentence for the idea, one for the skill. If the second one is blank, that is the part to revisit.

Discussion prompt

We fix one concrete network for the whole lesson: input dimension 2, a hidden layer of 3 ReLU neurons, and a single linear output. Every part of this deck comes back to it.

That is stated as though it always holds. Do one of two things: produce a case where it fails, or say precisely what rules such a case out. "It just does" is not on the menu.

Hint: Hunt at the extremes first — zero, one, negative, empty, equal. If every extreme survives, the reason they survive is the proof.

Discussion prompt

Explain Each hidden neuron is a little detector by analogy to something with no Machine Learning in it at all — a queue, a recipe, a map, a bank balance, whatever fits. Then say where your analogy breaks.

Hint: An analogy that never breaks is not an analogy, it is the same idea wearing a hat. Find the seam — that is the part that is actually new.

Put in order

Put the moves of Forward pass, step 1 — the pre-activations z⁽¹⁾ into the order they have to happen.

1. Neuron 0: (1)(1) + (−1)(2) + 0 = 1 − 2 = −1
2. Neuron 1: (0.5)(1) + (0.5)(2) − 1 = 0.5 + 1 − 1 = 0.5
3. Neuron 2: (−2)(1) + (1)(2) + 0.5 = −2 + 2 + 0.5 = 0.5

Why: These are the moves of the worked example in the order it makes them, and each one is set up by the one before it. Row 0 of W⁽¹⁾ is [1, −1], bias 0.

Comparison matrix

From Forward pass, step 1 — the pre-activations z⁽¹⁾: refill the row Wⱼ column from what you know. The rest of the table is as it appeared.

Predict first

Apply ReLU(z) = max(0, z) element-wise to z⁽¹⁾ = [−1, 0.5, 0.5]. ReLU clamps negatives to zero and leaves positives untouched:

Commit before you compute: what does Forward pass, step 2 — ReLU gives h⁽¹⁾ come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: Neurons 1, 2: max(0, 0.5) = 0.5 each → pass through

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. Positive pre-activations are unchanged, so h⁽¹⁾ = [0, 0.5, 0.5].

Comparison matrix

From Forward pass, step 2 — ReLU gives h⁽¹⁾: every row here is a choice with a cost. Fill the state column, then say which row you would actually pick and what you give up for it.

Discussion prompt

Output layer is linear: z⁽²⁾ = W⁽²⁾h⁽¹⁾ + b⁽²⁾ with W⁽²⁾ = [1, −1, 2], b⁽²⁾ = 0.5, and h⁽¹⁾ = [0, 0.5, 0.5]. One dot product:

What do you need to know — or decide — before the first line can be written? List everything the problem has to hand you.

Hint: Anything you would have to invent to get started is a thing the problem must supply.

Annotate

Walk the callouts on Forward pass, step 3 — the output z⁽²⁾. Each one is a place this is easy to get subtly wrong.

• Multiply each hidden output by its committee weight, sum, add the output bias.
• The dead neuron drops out, neuron 1 subtracts 0.5, neuron 2 adds 1.0, bias adds 0.5. Network output for x = [1, 2] is exactly 1.0.

Predict first

The table runs: hand trace (numpy) | 1.0 · nn.Sequential(torch) | 1.0

In Confirm the hand trace against PyTorch, given the rows so far: what is the next one — the row where source is match??

Correct: match? | yes

Why: The relationship between the columns, not the individual numbers, is what generates the next row. nn.Linear stores weight as (out, in) and computes xWᵀ + b, which is exactly W⁽¹⁾x + b⁽¹⁾ for our row layout.

Discussion prompt

Work backwards. The example finished here:

torch prints [1.] — identical to the hand trace

What was it asked to do, and what must it have been given? Reconstruct the problem from its answer.

Hint: Every quantity in the result had to enter somewhere. Account for each one.

Discussion prompt

Explain The universal approximation theorem to a student a year behind you. No notation, no jargon they have not met — and it still has to be true.

Hint: If your explanation needs a symbol they have never seen, you are describing the notation rather than the idea.

Sort into buckets

Sort these by target, from memory, without looking back at XOR: one hidden layer solves it. Telling them apart on the spot is the skill; the table is only where the answer happens to be written down.

0

(0,0); (1,1)

1

(0,1); (1,0)

g1

target is "0" for (0,0), (1,1) — that is what the table on "XOR: one hidden layer solves it" records, and it is the single property separating this group from the rest.

g2

target is "1" for (0,1), (1,0) — that is what the table on "XOR: one hidden layer solves it" records, and it is the single property separating this group from the rest.

Predict first

A student writes this, and it looks reasonable:

XOR just needs more capacity — drop the hidden layer and train a single Linear(2,1) harder.

It is wrong. Say what breaks — and say it before you turn the page.

Correct: A Linear + Sigmoid is one linear boundary.

Add a nonlinear hidden layer. Nonlinearity — not size — is what defeats non-separability.

Why: A Linear + Sigmoid is one linear boundary. No line separates XOR, so the best it can do is sit on the fence at 0.5 for all four points. More epochs never fix a representational limit.

Discussion prompt

The rule this trap just fixed:

Add a nonlinear hidden layer. Nonlinearity — not size — is what defeats non-separability.

Now break it on purpose. Build a case that violates it and follow the consequences until something visibly fails. Where does the failure first show up — and would you have noticed it if you had not been looking?

Hint: The dangerous rules are the ones whose violation still produces an answer. If yours fails loudly, try to find one that fails quietly.

Sort into buckets

These are the pieces of Lesson 44: MLP Architectures — Depth, Width & Activations, out of order. Put each one back under the part of the lesson it belongs to.

What an MLP actually computes

An MLP is stacked affine-then-nonlinear; Our running network: 2 → 3 → 1; Each hidden neuron is a little detector

Universal approximation — and its catch

The universal approximation theorem; The catch: 'enough' can be astronomically many; XOR: one hidden layer solves it

Depth vs width — why depth wins

Depth = composition; width = interpolation; Counting linear regions: depth is exponential; Composition reuses work; width repeats it

s1

What an MLP actually computes is where Lesson 44: MLP Architectures — Depth, Width & Activations puts An MLP is stacked affine-then-nonlinear, Our running network: 2 → 3 → 1, Each hidden neuron is a little detector. Knowing which part of the lesson a problem belongs to is most of knowing which method to reach for.

s2

Universal approximation — and its catch is where Lesson 44: MLP Architectures — Depth, Width & Activations puts The universal approximation theorem, The catch: 'enough' can be astronomically many, XOR: one hidden layer solves it. Knowing which part of the lesson a problem belongs to is most of knowing which method to reach for.

s3

Depth vs width — why depth wins is where Lesson 44: MLP Architectures — Depth, Width & Activations puts Depth = composition; width = interpolation, Counting linear regions: depth is exponential, Composition reuses work; width repeats it. Knowing which part of the lesson a problem belongs to is most of knowing which method to reach for.

Put in order

Put the moves of Params of our 2 → 3 → 1 network into the order they have to happen.

1. Layer 1 Linear(2, 3): 2×3 + 3 = 6 + 3 = 9
2. Layer 2 Linear(3, 1): 3×1 + 1 = 3 + 1 = 4
3. torch confirms total = 13

Why: These are the moves of the worked example in the order it makes them, and each one is set up by the one before it. 6 weights (the 3×2 matrix) plus 3 biases (one per hidden neuron).

Draw it

Draw what Params of our 2 → 3 → 1 network just did — the shape of it, not the line-by-line working. One picture, labels only where you need them. Then check it against the steps: anything you could not draw is a step you followed rather than understood.

Draw it

Draw what Params of a real MLP: [64 → 128 → 64 → 10] just did — the shape of it, not the line-by-line working. One picture, labels only where you need them. Then check it against the steps: anything you could not draw is a step you followed rather than understood.

Predict first

A student writes this, and it looks reasonable:

Linear(64, 128) is a 64×128 weight matrix, so 64 × 128 = 8192 parameters.

It is wrong. Say what breaks — and say it before you turn the page.

Correct: Misses the 128-element bias vector.

Add the bias: d_in × d_out + d_out = 64×128 + 128 = 8320.

Why: Misses the 128-element bias vector. That is an off-by-d_out error on EVERY layer; across a deep net it compounds into thousands of uncounted parameters and a wrong size budget.

Annotate

Walk the callouts on Trap: forgetting the bias in the count. Each one is a place this is easy to get subtly wrong.

• Misses the 128-element bias vector. That is an off-by-d_out error on EVERY layer; across a deep net it compounds into thousands of uncounted parameters and a wrong size budget.
• The bias adds one scalar per output neuron (128 of them). Verified: sum(p.numel() for p in Linear(64,128).parameters()) = 8320. Always add d_out.

Predict first

Evaluate all four at x ∈ {−2, −1, 0, 1, 2}, and also the sigmoid derivative σ'(x) = σ(x)(1−σ(x)) — the quantity that governs gradient flow:

Commit before you compute: what does Exact activation values — computed come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: σ'(±2) = 0.1050 already; the max of σ' is only 0.25 (at x=0)

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. The sigmoid derivative never exceeds 0.25 anywhere, and shrinks fast toward the tails.

Sort into buckets

Sort these by relu, from memory, without looking back at Exact activation values — computed. Telling them apart on the spot is the skill; the table is only where the answer happens to be written down.

0.0000

−2.0; −1.0; 0.0

1.0000

1.0

2.0000

2.0

g1

relu is "0.0000" for −2.0, −1.0, 0.0 — that is what the table on "Exact activation values — computed" records, and it is the single property separating this group from the rest.

g2

relu is "1.0000" for 1.0 — that is what the table on "Exact activation values — computed" records, and it is the single property separating this group from the rest.

g3

relu is "2.0000" for 2.0 — that is what the table on "Exact activation values — computed" records, and it is the single property separating this group from the rest.

Fill in the blanks

From GELU from scratch — verified against nn.GELU — one line has had its right-hand side removed. Put it back.

import numpy as np, torch, torch.nn as nn
from scipy.special import erf
def gelu_scratch(x):
return x * 0.5 * (1.0 + erf(x / np.sqrt(2)))
pts = np.array([-1., 0., 1., 2.])
manual = gelu_scratch(pts)
torch_v = nn.GELU()(torch.tensor(pts, dtype=torch.float32)).numpy()
for x, m, t in zip(pts, manual, torch_v):
print(f'x=___ manual=___ torch=___ match=___')

Why: manual is what everything below it consumes, so the wrong expression here fails later and somewhere else. GELU(−1)=−0.158655, GELU(0)=0, GELU(1)=0.841345, GELU(2)=1.954500.

Step through it

Step through GELU from scratch — verified against nn.GELU one row at a time. One of these columns never changes — find it, and say why it cannot.

1. Step 1: x is −1.0
2. Step 2: x is 0.0
3. Step 3: x is 1.0
4. Step 4: x is 2.0

Discussion prompt

Explain Failure mode 1: vanishing gradients to a student a year behind you. No notation, no jargon they have not met — and it still has to be true.

Hint: If your explanation needs a symbol they have never seen, you are describing the notation rather than the idea.

Comparison matrix

From Quantify the vanishing gradient: refill the 0.25ᴸ (best) column from what you know. The rest of the table is as it appeared.

Predict first

The table runs: 0 | −5.164 | 0.0 | 0 · 1 | −6.220 | 0.0 | 0 · 2 | −3.890 | 0.0 | 0

In Failure mode 2: the dying ReLU, given the rows so far: what is the next one — the row where sample is … (all 6)?

Correct: … (all 6) | all < 0 | 0.0 | 0 (dead)

Why: The relationship between the columns, not the individual numbers, is what generates the next row. preacts ≈ [−5.16, −6.22, −3.89, −3.80, −3.31, −4.60], all < 0, so the gradient gate is 0 on every sample.

Step through it

Step through Failure mode 2: the dying ReLU one row at a time. One of these columns never changes — find it, and say why it cannot.

1. Step 1: sample is 0
2. Step 2: sample is 1
3. Step 3: sample is 2
4. Step 4: sample is … (all 6)

Predict first

A student writes this, and it looks reasonable:

Sigmoid is a classic activation — use it in every hidden layer so activations stay in (0, 1).

It is wrong. Say what breaks — and say it before you turn the page.

Correct: Each backprop step multiplies by σ'(x) ≤ 0.25.

Use ReLU or GELU in hidden layers; reserve sigmoid for the output of binary classification, where saturation is the desired probability clamp.

Why: Each backprop step multiplies by σ'(x) ≤ 0.25. After 12 layers the gradient is ≤ 0.25¹² ≈ 6×10⁻⁸ — the early layers get no signal and training freezes.

Sort into buckets

Some of these hold up and some are the exact mistakes this lesson is built to prevent. Sort them.

Holds up

We feed it the input x = [1, 2]. Row j of W⁽¹⁾ holds the two weights of hidden neuron j. Keep this table of numbers in view — the next four slides trace it end to end.; The output layer is a weighted committee of those detectors. Learning means tuning who votes, how strongly each detector responds, and how the committee combines them.; Rule of thumb: hierarchical targets (images, sequences) reward depth; a smooth scalar target with no hierarchy is fine with a moderately wide shallow net.

Breaks

XOR just needs more capacity — drop the hidden layer and train a single Linear(2,1) harder.; Linear(64, 128) is a 64×128 weight matrix, so 64 × 128 = 8192 parameters.

sound

These are stated as this lesson states them — each one survives the edge cases Lesson 44: MLP Architectures — Depth, Width & Activations puts it through.

flawed

Each of these is lifted from a trap in this deck: reasonable-sounding, and wrong in a way that only shows up once you rely on it.

Discussion prompt

A fair comparison holds the parameter count fixed and varies only the shape. Compute the budgets by hand: a wide net 64→512→10 vs a deep net (9 hidden layers of width 64, then 64→10).

What do you need to know — or decide — before the first line can be written? List everything the problem has to hand you.

Hint: Anything you would have to invent to get started is a thing the problem must supply.

Predict first

Train both on load_digits (1 797 samples, 64 features, 10 classes), same optimizer, same seed, same 400 epochs. This block is self-contained:

Commit before you compute: what does Wide vs deep on load_digits come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: wide → 97.50% test acc; deep → 94.72% test acc

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. On flat tabular digits the wide net WINS.

Discussion prompt

Work backwards. The example finished here:

wide → 97.50% test acc; deep → 94.72% test acc

What was it asked to do, and what must it have been given? Reconstruct the problem from its answer.

Hint: Every quantity in the result had to enter somewhere. Account for each one.

Discussion prompt

Explain USAAIO architecture-design guidelines by analogy to something with no Machine Learning in it at all — a queue, a recipe, a map, a bank balance, whatever fits. Then say where your analogy breaks.

Hint: An analogy that never breaks is not an analogy, it is the same idea wearing a hat. Find the seam — that is the part that is actually new.

Discussion prompt

Run The MLP architecture recipe with this step confiscated:

Output activation: none/logits for CrossEntropyLoss · sigmoid for binary · softmax only if you need explicit probabilities

Is it still possible? If it is, say what takes its place and what it costs you. If it is not, say exactly what that step was providing that nothing else does.

Hint: A step you can drop for free was never load-bearing. If you cannot drop it, name the thing that goes wrong the moment it is gone.

Discussion prompt

The MLP architecture recipe works on the cases you have just seen. Push it to the edge: what is the most degenerate input it still handles — empty, zero, one item, everything equal — and what is the first case where it stops being true? Name the case, not just "it breaks".

Hint: Try the smallest legal input, then the largest, then the one where two things collide. Methods are specified at their edges; the middle takes care of itself.

Eliminate the wrong options

With W⁽²⁾ = [1, −1, 2], b⁽²⁾ = 0.5, and hidden output h⁽¹⁾ = [0, 0.5, 0.5], what is the network's output?

3 of these 4 are wrong. Strike them one at a time, and say what rules each one out before you strike the next. The survivor is the answer.

• A. 1.0
• B. 0.5
• C. 2.0
• D. 1.5

Survives elimination: A

Why: z⁽²⁾ = (1)(0) + (−1)(0.5) + (2)(0.5) + 0.5 = 0 − 0.5 + 1.0 + 0.5 = 1.0. The dead neuron 0 drops out; neuron 1 subtracts 0.5, neuron 2 adds 1.0, and the bias adds 0.5.

Check your understanding

With W⁽²⁾ = [1, −1, 2], b⁽²⁾ = 0.5, and hidden output h⁽¹⁾ = [0, 0.5, 0.5], what is the network's output?

• A. 1.0 (correct)
• B. 0.5
• C. 2.0
• D. 1.5

Answer: A

Why: z⁽²⁾ = (1)(0) + (−1)(0.5) + (2)(0.5) + 0.5 = 0 − 0.5 + 1.0 + 0.5 = 1.0. The dead neuron 0 drops out; neuron 1 subtracts 0.5, neuron 2 adds 1.0, and the bias adds 0.5.

Why B tempts people

This is just the bias b⁽²⁾ = 0.5 — it forgets the two firing hidden neurons' contributions (−0.5 and +1.0), which net to +0.5 on top of the bias.

Why C tempts people

Uses the raw pre-activations z⁽¹⁾ instead of the ReLU outputs, or forgets neuron 0 was clamped to 0 — double-counting the silenced neuron.

Why D tempts people

Adds the two firing outputs as +0.5 and +1.0 (both positive), ignoring that neuron 1's committee weight is −1, so its contribution is −0.5, not +0.5.

Predict first

How many trainable parameters does nn.Linear(128, 64) have?

Answer it in your own words, now, with nothing to choose from. The options are on the next slide — and picking the right one off a list is an easier skill than producing it.

Correct: 8 256 (128×64 + 64)

Why: d_in × d_out + d_out = 128×64 + 64 = 8192 + 64 = 8256. The +64 is the bias, one scalar per output neuron. Verified: sum(p.numel() for p in nn.Linear(128,64).parameters()) = 8256.

Check your understanding

How many trainable parameters does nn.Linear(128, 64) have?

• A. 8 256 (128×64 + 64) (correct)
• B. 8 192 (128×64 only)
• C. 8 320 (128×64 + 128)
• D. 192 (128 + 64)

Answer: A

Why: d_in × d_out + d_out = 128×64 + 64 = 8192 + 64 = 8256. The +64 is the bias, one scalar per output neuron. Verified: sum(p.numel() for p in nn.Linear(128,64).parameters()) = 8256.

Why B tempts people

Counts only the weight matrix (128×64 = 8192) and forgets the 64-element bias vector — the classic off-by-bias error.

Why C tempts people

Adds d_in (128) instead of d_out (64) as the bias count. The bias length matches the OUTPUT dimension, not the input.

Why D tempts people

Adds the two sizes as if params were additive. The weight matrix is multiplicative (128×64); only the bias is additive (+64).

Predict first

You build a 12-layer MLP. Which hidden-layer activation is most likely to stall training because gradients vanish?

Answer it in your own words, now, with nothing to choose from. The options are on the next slide — and picking the right one off a list is an easier skill than producing it.

Correct: sigmoid — derivative ≤ 0.25 compounds to near-zero across layers

Why: σ'(x) ≤ 0.25 everywhere. Across 12 layers the gradient into the first layer is ≤ 0.25¹² ≈ 6×10⁻⁸ — effectively zero. ReLU's gradient is 1 for active neurons, so it does not compound.

Check your understanding

You build a 12-layer MLP. Which hidden-layer activation is most likely to stall training because gradients vanish?

• A. sigmoid — derivative ≤ 0.25 compounds to near-zero across layers (correct)
• B. ReLU — outputs zero for negative inputs
• C. GELU — its smooth curve attenuates the signal
• D. tanh — its range (−1, 1) is too small

Answer: A

Why: σ'(x) ≤ 0.25 everywhere. Across 12 layers the gradient into the first layer is ≤ 0.25¹² ≈ 6×10⁻⁸ — effectively zero. ReLU's gradient is 1 for active neurons, so it does not compound.

Why B tempts people

ReLU does zero out negatives (and dead neurons are real), but its gradient for ACTIVE neurons is exactly 1 — no decay compounds across depth.

Why C tempts people

GELU is smooth and never hard-zeros; its derivative is near 1 for positive inputs, so it does not vanish across depth the way sigmoid does.

Why D tempts people

tanh's range is (−1,1) but its derivative at 0 is 1 and it decays less steeply than sigmoid; range size is not what causes vanishing gradients — the small derivative is.

Eliminate the wrong options

On load_digits at a ~38k parameter budget, the wide model (1 hidden, width 512) hit 97.5% and the deep model (10 hidden, width 64) hit 94.7%. Best interpretation?

3 of these 4 are wrong. Strike them one at a time, and say what rules each one out before you strike the next. The survivor is the answer.

• A. Width won here because digits is flat/tabular; depth helps mainly on hierarchical data
• B. Depth always beats width at equal parameter count
• C. The deep model overfit because it had more parameters
• D. It proves sigmoid is the better activation for digits

Survives elimination: A

Why: load_digits is a flat 64-feature problem with no spatial hierarchy, so a single wide layer interpolates efficiently and the deep ReLU stack is slightly harder to optimize. Depth shines on hierarchical data (images, sequences) — the result is dataset-dependent.

Check your understanding

On load_digits at a ~38k parameter budget, the wide model (1 hidden, width 512) hit 97.5% and the deep model (10 hidden, width 64) hit 94.7%. Best interpretation?

• A. Width won here because digits is flat/tabular; depth helps mainly on hierarchical data (correct)
• B. Depth always beats width at equal parameter count
• C. The deep model overfit because it had more parameters
• D. It proves sigmoid is the better activation for digits

Answer: A

Why: load_digits is a flat 64-feature problem with no spatial hierarchy, so a single wide layer interpolates efficiently and the deep ReLU stack is slightly harder to optimize. Depth shines on hierarchical data (images, sequences) — the result is dataset-dependent.

Why B tempts people

This experiment is a direct counterexample. Depth wins when the target has compositional/hierarchical structure — not universally.

Why C tempts people

The deep model actually had FEWER parameters (38 090 vs 38 410), and overfitting would show low train loss with high test loss; both converged to near-zero train loss.

Why D tempts people

Both models used ReLU; sigmoid was never tested here and is irrelevant to the accuracy comparison.

Discussion prompt

Build MLPs of different shapes on load_digits, count their parameters exactly, and reimplement GELU from the formula — proving every number against torch and scipy.

That is stated as though it always holds. Do one of two things: produce a case where it fails, or say precisely what rules such a case out. "It just does" is not on the menu.

Hint: Hunt at the extremes first — zero, one, negative, empty, equal. If every extreme survives, the reason they survive is the proof.

Step through it

Step through Milestone 2 — GELU from scratch one row at a time. One of these columns never changes — find it, and say why it cannot.

1. Step 1: x is −1.0
2. Step 2: x is 0.0
3. Step 3: x is 1.0
4. Step 4: x is 2.0

Comparison matrix

From Milestone 3 — train and score: every row here is a choice with a cost. Fill the value (verified) column, then say which row you would actually pick and what you give up for it.

Comparison matrix

From The full program: refill the value column from what you know. The rest of the table is as it appeared.

Draw it

One page, no notation unless you need it: draw how these connect — What an MLP actually computes · Universal approximation — and its catch · Depth vs width — why depth wins · Parameter counting — exact formula · Activation functions — outputs & gradients · Deep vs wide — a fair experiment. Put an arrow wherever one of them is what makes another possible, and label the arrow with why.