Lesson 36: Initialization & Gradient Clipping

Discussion prompt

Before we open Lesson 36: Initialization & Gradient Clipping: without looking back, what was the main idea of ROC, PR Curves & Calibration, and what could you do by the end of it that you could not do before?

Hint: One sentence for the idea, one for the skill. If the second one is blank, that is the part to revisit.

Discussion prompt

A feed-forward net stacks L layers: each takes the previous activation a, multiplies by a weight matrix W, and applies a nonlinearity. The output is a long composition of these maps.

That is stated as though it always holds. Do one of two things: produce a case where it fails, or say precisely what rules such a case out. "It just does" is not on the menu.

Hint: Hunt at the extremes first — zero, one, negative, empty, equal. If every extreme survives, the reason they survive is the proof.

Discussion prompt

Explain Why 'a little off' becomes 'catastrophic' by analogy to something with no Machine Learning in it at all — a queue, a recipe, a map, a bank balance, whatever fits. Then say where your analogy breaks.

Hint: An analogy that never breaks is not an analogy, it is the same idea wearing a hat. Find the seam — that is the part that is actually new.

Predict first

Make the per-layer factor concrete. Take 30 layers, each multiplying the gradient magnitude by a constant c. The overall factor is c^30. Three cases — this snippet is complete and runnable:

Commit before you compute: what does Depth turns 1.5 into 190,000× come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: c = 1.5 → 1.9e+05 (explodes); c = 0.7 → 2.3e-05 (vanishes); c = 1.0 → 1.0 (holds)

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. A 50% per-layer over-scale becomes a 190,000× blow-up in 30 layers; a 30% under-scale shrinks the gradient to 2.3e-5.

Comparison matrix

From Depth turns 1.5 into 190,000×: refill the c^30 (verified) column from what you know. The rest of the table is as it appeared.

Sort into buckets

Every line below is part of the definition of exploding gradients or of vanishing gradients — one or the other, never both. Put each where it belongs.

exploding gradients

The per-layer factor is > 1, so the gradient product grows exponentially with depth; huge parameter jumps, inf/NaN losses, divergence.; Common in deep and recurrent nets.

vanishing gradients

The per-layer factor is < 1, so the gradient product decays exponentially; early layers receive a gradient near 0 and stop learning.; The quieter, more insidious failure.

b1

The per-layer factor is > 1, so the gradient product grows exponentially with depth — huge parameter jumps, inf/NaN losses, divergence. Common in deep and recurrent nets.

b2

The per-layer factor is < 1, so the gradient product decays exponentially — early layers receive a gradient near 0 and stop learning. The quieter, more insidious failure.

Discussion prompt

Explain Two separate fixes to a student a year behind you. No notation, no jargon they have not met — and it still has to be true.

Hint: If your explanation needs a symbol they have never seen, you are describing the notation rather than the idea.

Put in order

Put the moves of Derive the clip factor into the order they have to happen.

1. Only act when the gradient is too long
2. Scale by a single positive number s so the new norm equals τ
3. Fold both cases into one min()

Why: These are the moves of the worked example in the order it makes them, and each one is set up by the one before it. If ‖g‖ ≤ τ the step is already safe — leave g untouched.

Annotate

From Derive the clip factor — read this one piece at a time. What is each part doing?

On: \( \lVert s\,g \rVert = s\,\lVert g \rVert = \tau \;\Longrightarrow\; s = \frac{\tau}{\lVert g \rVert} \)

• If ‖g‖ ≤ τ the step is already safe — leave g untouched. We only rescale when ‖g‖ > τ.
• Multiplying g by s > 0 keeps its direction; we choose s so ‖s·g‖ = τ.
• When ‖g‖ ≤ τ we want s = 1; when ‖g‖ > τ we want s = τ/‖g‖. Since τ/‖g‖ ≥ 1 exactly when ‖g‖ ≤ τ, the minimum of the two picks the right branch automatically.

Discussion prompt

Take g = [30, 40], threshold τ = 5. Its norm is √(30² + 40²) = √2500 = 50, so s = 5/50 = 0.1. Every number below is real output:

What do you need to know — or decide — before the first line can be written? List everything the problem has to hand you.

Hint: Anything you would have to invent to get started is a thing the problem must supply.

Comparison matrix

From Clip by norm — the trace: every row here is a choice with a cost. Fill the value (verified) column, then say which row you would actually pick and what you give up for it.

Predict first

A student writes this, and it looks reasonable:

To cap the gradient, just clamp each entry into [−τ, τ] with np.clip(g, -tau, tau).

It is wrong. Say what breaks — and say it before you turn the page.

Correct: Original unit direction was [0.6, 0.8].

Scale the whole vector by one factor s = min(1, τ/‖g‖).

Why: Original unit direction was [0.6, 0.8]. Clamping capped 30 and 40 to the SAME 5, so the ratio changed and the vector turned 8° off. You're no longer descending the loss — verified: keeps-direction = False.

Predict first

The table runs: original g | [30, 40] | [0.6, 0.8] | — · norm clip (×0.1) | [3, 4] | [0.6, 0.8] | True

In Side by side: norm clip vs clamp, given the rows so far: what is the next one — the row where method is per-component clamp?

Correct: per-component clamp | [5, 5] | [0.707, 0.707] | False

Why: The relationship between the columns, not the individual numbers, is what generates the next row. The two methods produce genuinely different directions.

Discussion prompt

Work backwards. The example finished here:

norm-clip dir = [0.6, 0.8] (True); clamp dir = [0.707, 0.707] (False)

What was it asked to do, and what must it have been given? Reconstruct the problem from its answer.

Hint: Every quantity in the result had to enter somewhere. Account for each one.

Sort into buckets

These are the pieces of Lesson 36: Initialization & Gradient Clipping, out of order. Put each one back under the part of the lesson it belongs to.

The problem: depth multiplies

A deep net is a chain of maps; The gradient is a product; Why 'a little off' becomes 'catastrophic'

Clipping exploding gradients

The idea: cap the length, keep the aim; Derive the clip factor; Clip by norm — the trace

Variance through a layer

Track the signal's 'loudness'; Setup: one linear layer; Why variance, not the mean

s1

The problem: depth multiplies is where Lesson 36: Initialization & Gradient Clipping puts A deep net is a chain of maps, The gradient is a product, Why 'a little off' becomes 'catastrophic'. Knowing which part of the lesson a problem belongs to is most of knowing which method to reach for.

s2

Clipping exploding gradients is where Lesson 36: Initialization & Gradient Clipping puts The idea: cap the length, keep the aim, Derive the clip factor, Clip by norm — the trace. Knowing which part of the lesson a problem belongs to is most of knowing which method to reach for.

s3

Variance through a layer is where Lesson 36: Initialization & Gradient Clipping puts Track the signal's 'loudness', Setup: one linear layer, Why variance, not the mean. Knowing which part of the lesson a problem belongs to is most of knowing which method to reach for.

Discussion prompt

Variance of one output entry — before any calculation: what is the plan? Name the moves in order, in plain English, without doing the arithmetic.

Hint: It starts with: Variance of a sum of independent terms adds

Draw it

Translate both ways. First write the expression above as a sentence with no symbols in it at all. Then cover it, and write your sentence back as notation. If the two versions disagree, the disagreement is the thing to fix.

Fill in the blanks

From Confirm Var(z) = n·Var(W)·Var(a) — one line has had its right-hand side removed. Put it back.

import numpy as np
rng = np.random.default_rng(0)
n = 512; sigma2 = 1.0 / n
zs = []
for _ in range(4000):
a = rng.normal(size=n) # Var(a) = 1
w = rng.normal(size=n) * np.sqrt(sigma2)
zs.append(w @ a)
print('predicted:', n * sigma2 * 1.0)
print('empirical :', round(float(np.var(zs)), 4))

Why: n is what everything below it consumes, so the wrong expression here fails later and somewhere else. The tiny gap is Monte-Carlo noise from 4000 samples.

Fill in the blanks

The ratio predicts the 50-layer std, with the scaffolding fading: two lines are gone now — fill both.

n = 256
for name, scale in [('He', 2), ('Xavier', 1), ('too_big', 8)]:
varW = scale / n
r = **0.5 * n * varW** # ReLU per-layer variance ratio
print(f'___: r=___, Var ratio r^50 = ___')

Why: Reproducing these unaided, rather than reading them, is what tells you the method has transferred. He's ½·256·(2/256) = 1 is the ONLY ratio that survives depth.

Put in order

Put the moves of Solve for Var(W): the forward condition into the order they have to happen.

1. Cancel Var(a) from both sides
2. Isolate Var(W)
3. Backward gives the mirror condition

Why: These are the moves of the worked example in the order it makes them, and each one is set up by the one before it. Var(a) > 0, so divide it out. The activation variance drops away, leaving a pure condition on W and the fan-in.

Draw it

Draw what Solve for Var(W): the forward condition just did — the shape of it, not the line-by-line working. One picture, labels only where you need them. Then check it against the steps: anything you could not draw is a step you followed rather than understood.

Predict first

A student writes this, and it looks reasonable:

Xavier is 'the 2/(n_in+n_out) one', so on a square layer with n_in = n_out = n its variance is 2/n.

It is wrong. Say what breaks — and say it before you turn the page.

Correct: Plugging n_in = n_out = n into 2/(n_in+n_out) gives 2/(n+n) = 2/(2n) = 1/n, NOT 2/n.

Substitute carefully: the denominator is n_in + n_out = 2n.

Why: Plugging n_in = n_out = n into 2/(n_in+n_out) gives 2/(n+n) = 2/(2n) = 1/n, NOT 2/n. Forgetting to add the two n's in the denominator doubles the variance and starts the signal growing.

Discussion prompt

The rule this trap just fixed:

Substitute carefully: the denominator is n_in + n_out = 2n.

Now break it on purpose. Build a case that violates it and follow the consequences until something visibly fails. Where does the failure first show up — and would you have noticed it if you had not been looking?

Hint: The dangerous rules are the ones whose violation still produces an answer. If yours fails loudly, try to find one that fails quietly.

Discussion prompt

Explain ReLU throws away half the signal to a student a year behind you. No notation, no jargon they have not met — and it still has to be true.

Hint: If your explanation needs a symbol they have never seen, you are describing the notation rather than the idea.

Predict first

For symmetric mean-zero z, relu(z) = max(0, z) keeps the positive half untouched and zeros the rest. Claim: E[relu(z)²] = ½·Var(z). Measure it on 2 million draws:

Commit before you compute: what does ReLU keeps exactly half the variance come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: E[relu(z)²] ≈ 0.5006 ≈ ½·Var(z)

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. Half the z's are zeroed (contribute nothing); the surviving half carry the full second moment, and by symmetry that's exactly half the total.

Put in order

Put the moves of Derive He: put the half back into the order they have to happen.

1. Forward relation with a ReLU layer
2. Demand the activation variance holds
3. Solve for Var(W) — the factor of 2 appears

Why: These are the moves of the worked example in the order it makes them, and each one is set up by the one before it. The pre-activation still has Var(z) = n·Var(W)·Var(a).

Annotate

From Derive He: put the half back — read this one piece at a time. What is each part doing?

On: \( \boxed{\;\mathrm{Var}(W)_{\text{He}} = \frac{2}{n_{\text{in}}}\;} \)

• The pre-activation still has Var(z) = n·Var(W)·Var(a). But the activation a = relu(z) now carries only HALF that variance, from the previous slide.
• Set Var(a^(ℓ)) = Var(a^(ℓ-1)) so the signal is stable through depth, then cancel that common variance.
• Multiply both sides by 2 and divide by n. The 2 is exactly the compensation for ReLU zeroing half the units.

Predict first

A student writes this, and it looks reasonable:

Xavier is the classic initializer, so use it everywhere — including deep ReLU nets.

It is wrong. Say what breaks — and say it before you turn the page.

Correct: Each ReLU halves the variance and Xavier's 1/n only preserves the LINEAR part, so the activation loses ~half its power per layer.

Use He (2/n_in) for ReLU — the factor of 2 refills the deleted half each layer.

Why: Each ReLU halves the variance and Xavier's 1/n only preserves the LINEAR part, so the activation loses ~half its power per layer. Verified: std decays to 1.47e-08 by layer 50 — the signal is gone.

Fill in the blanks

From He stabilizes the backward gradient too — one line has had its right-hand side removed. Put it back.

import numpy as np
def grad_std(scale, depth=50, w=256, seed=1):
rng = np.random.default_rng(seed)
a = rng.normal(size=w); Ws, masks = [], []
for _ in range(depth): # forward, cache masks
W = rng.normal(size=(w, w)) * np.sqrt(scale / w)
z = W @ a; m = (z > 0).astype(float)
a = z * m; Ws.append(W); masks.append(m)
d = rng.normal(size=w) # backward from output
for L in range(depth - 1, -1, -1):
d = (d * masks[L]); d = Ws[L].T @ d
return d.std()
for name, s in [('He', 2), ('Xavier', 1)]:
print(name, f'___')

Why: d is what everything below it consumes, so the wrong expression here fails later and somewhere else. The backward pass sees the same halving from the ReLU mask (line 8) and the same Wᵀ scaling.

Fill in the blanks

From 50 ReLU layers, four initializations — one line has had its right-hand side removed. Put it back.

import numpy as np
def final_std(scale, depth=50, w=256, seed=0):
rng = np.random.default_rng(seed)
a = rng.normal(size=w)
for _ in range(depth):
W = rng.normal(size=(w, w)) * np.sqrt(scale / w)
a = np.maximum(0, W @ a) # ReLU
return a.std()
for name, s in [('He', 2), ('Xavier', 1),
('too_big', 8), ('too_small', 0.1)]:
print(name, f'___')

Why: a is what everything below it consumes, so the wrong expression here fails later and somewhere else. Only He's factor-of-2 keeps the std O(1) across 50 ReLU layers.

Comparison matrix

From 50 ReLU layers, four initializations: refill the Var(W) scale column from what you know. The rest of the table is as it appeared.

Predict first

The table runs: 1 | 8.54e-01 | 6.04e-01 · 2 | 7.98e-01 | 3.99e-01 · 5 | 8.97e-01 | 1.59e-01 · 10 | 9.26e-01 | 2.89e-02 · 25 | 7.58e-01 | 1.31e-04

In Watch it happen layer by layer, given the rows so far: what is the next one — the row where layer is 50?

Correct: 50 | 4.93e-01 | 1.47e-08

Why: The relationship between the columns, not the individual numbers, is what generates the next row. The full trace shows Xavier is losing roughly half its variance (≈0.7× std) each layer while He holds.

Step through it

Step through Watch it happen layer by layer one row at a time. What is driving the change, and what would the row after the last one be?

1. Step 1: layer is 1
2. Step 2: layer is 2
3. Step 3: layer is 5
4. Step 4: layer is 10
5. Step 5: layer is 25
6. Step 6: layer is 50

Discussion prompt

Explain Zero feels safe — and never learns by analogy to something with no Machine Learning in it at all — a queue, a recipe, a map, a bank balance, whatever fits. Then say where your analogy breaks.

Hint: An analogy that never breaks is not an analogy, it is the same idea wearing a hat. Find the seam — that is the part that is actually new.

Discussion prompt

Tiny net: 2 inputs → 3 hidden (ReLU) → 1 output, all weights zero, MSE loss. Forward is all zeros; backward gives identical rows in dW1. Runnable as written:

What do you need to know — or decide — before the first line can be written? List everything the problem has to hand you.

Hint: Anything you would have to invent to get started is a thing the problem must supply.

Predict first

Same net, but draw W from a He normal. Now the three neurons see different pre-activations, ReLU keeps a different subset alive, and the gradient rows differ. Runnable:

Commit before you compute: what does Random init breaks the symmetry come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: Neuron 0's pre-activation is negative → ReLU kills it; rows of dW1 differ

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. h_pre = [-0.139, 0.850, 0.188]: the first unit is off (gradient row is zero), the others are on with different gradients.

Discussion prompt

Work backwards. The example finished here:

Neuron 0's pre-activation is negative → ReLU kills it; rows of dW1 differ

What was it asked to do, and what must it have been given? Reconstruct the problem from its answer.

Hint: Every quantity in the result had to enter somewhere. Account for each one.

Predict first

A student writes this, and it looks reasonable:

Zero is a neutral, unbiased starting point, so set all weights to 0.

It is wrong. Say what breaks — and say it before you turn the page.

Correct: Identical weights ⇒ identical outputs ⇒ identical gradients ⇒ identical updates, forever.

Initialize randomly from the He (or Xavier) distribution to break symmetry.

Why: Identical weights ⇒ identical outputs ⇒ identical gradients ⇒ identical updates, forever. The layer collapses to one effective neuron and never learns distinct features. Bias terms alone can't break it — verified: dW1 rows all equal.

Sort into buckets

Some of these hold up and some are the exact mistakes this lesson is built to prevent. Sort them.

Holds up

Addition forgives small errors; multiplication does not. A per-layer factor of 1.5 sounds harmless, but you apply it 30 times, and 1.5 compounds like interest.; Both come from the same multiplicative product. Fix the per-layer factor and you fix both at once — which is exactly what initialization does.; Clipping treats the symptom; initialization treats the cause. Part 1 does clipping — it's the quick win — then Parts 2–6 build initialization from variance.

Breaks

To cap the gradient, just clamp each entry into [−τ, τ] with np.clip(g, -tau, tau).; Xavier is 'the 2/(n_in+n_out) one', so on a square layer with n_in = n_out = n its variance is 2/n.

sound

These are stated as this lesson states them — each one survives the edge cases Lesson 36: Initialization & Gradient Clipping puts it through.

flawed

Each of these is lifted from a trap in this deck: reasonable-sounding, and wrong in a way that only shows up once you rely on it.

Put in order

These are the steps of The init / stability toolkit, scrambled. Put them back in order before the next slide shows you.

1. Exploding grads: clip by norm g·min(1, τ/‖g‖) — caps length, keeps direction (never clamp per component)
2. ReLU nets: He Var(W) = 2/n_in — the 2 refills the half ReLU deletes
3. tanh / sigmoid: Xavier Var(W) = 2/(n_in+n_out) — preserves variance for symmetric activations
4. Never init all weights equal — random draw breaks neuron symmetry
5. Diagnose by printing per-layer activation and gradient std: it should stay O(1), not drift toward 0 or ∞

Why: This is the order the recipe itself gives. Recalling the sequence without the slide in front of you is the difference between recognising the method and being able to run it — most of what goes wrong in practice is a step done out of turn.

Discussion prompt

The init / stability toolkit works on the cases you have just seen. Push it to the edge: what is the most degenerate input it still handles — empty, zero, one item, everything equal — and what is the first case where it stops being true? Name the case, not just "it breaks".

Hint: Try the smallest legal input, then the largest, then the one where two things collide. Methods are specified at their edges; the middle takes care of itself.

Eliminate the wrong options

Why do gradients explode or vanish specifically in DEEP networks?

3 of these 4 are wrong. Strike them one at a time, and say what rules each one out before you strike the next. The survivor is the answer.

• A. The gradient is a product of per-layer factors, so a factor slightly off 1 compounds exponentially with depth
• B. Deep networks use more memory, which rounds the gradients to 0
• C. ReLU is undefined at 0, and deeper nets hit that point more often
• D. The learning rate must grow with depth, overshooting the minimum

Survives elimination: A

Why: By the chain rule the gradient at an early layer is a product of one factor per later layer. A product of L numbers near c behaves like c^L: c = 1.5 gives 1.9e5 over 30 layers, c = 0.7 gives 2.3e-5. Only c ≈ 1 survives — which is what initialization targets.

Check your understanding

Why do gradients explode or vanish specifically in DEEP networks?

• A. The gradient is a product of per-layer factors, so a factor slightly off 1 compounds exponentially with depth (correct)
• B. Deep networks use more memory, which rounds the gradients to 0
• C. ReLU is undefined at 0, and deeper nets hit that point more often
• D. The learning rate must grow with depth, overshooting the minimum

Answer: A

Why: By the chain rule the gradient at an early layer is a product of one factor per later layer. A product of L numbers near c behaves like c^L: c = 1.5 gives 1.9e5 over 30 layers, c = 0.7 gives 2.3e-5. Only c ≈ 1 survives — which is what initialization targets.

Why B tempts people

Memory has nothing to do with it — the effect is purely the multiplicative chain-rule product, and it happens in exact arithmetic too.

Why C tempts people

ReLU's kink at 0 is a measure-zero event and isn't the cause; the compounding product of Jacobian factors is.

Why D tempts people

The learning rate is a fixed hyperparameter, not something that grows with depth; the instability comes from the gradient product itself, before the step size is applied.

Predict first

For a deep network with ReLU activations, the correct initialization variance (square layer, fan-in n) is:

Answer it in your own words, now, with nothing to choose from. The options are on the next slide — and picking the right one off a list is an easier skill than producing it.

Correct: He: Var(W) = 2/n

Why: ReLU zeroes ~half the activations (E[relu²] = ½·Var(z), verified ≈ 0.5006), so He's factor-of-2 variance refills the deleted half and keeps the signal stable — std ≈ 0.49 at layer 50. Xavier's 1/n under-scales for ReLU and the signal decays to 1.47e-8.

Check your understanding

For a deep network with ReLU activations, the correct initialization variance (square layer, fan-in n) is:

• A. He: Var(W) = 2/n (correct)
• B. Xavier: Var(W) = 1/n
• C. all weights = 0
• D. all weights = 1

Answer: A

Why: ReLU zeroes ~half the activations (E[relu²] = ½·Var(z), verified ≈ 0.5006), so He's factor-of-2 variance refills the deleted half and keeps the signal stable — std ≈ 0.49 at layer 50. Xavier's 1/n under-scales for ReLU and the signal decays to 1.47e-8.

Why B tempts people

Xavier (1/n on a square layer) preserves variance for symmetric activations; ReLU deletes an extra half each layer, so 1/n under-scales by 2× and activations decay to 1.47e-8 by layer 50.

Why C tempts people

Zero init sets the right variance to nothing AND breaks symmetry-breaking — every neuron stays identical and nothing learns; it's not a variance choice at all.

Why D tempts people

Constant nonzero init has the same broken-symmetry problem as zeros AND blows the forward signal up; a constant is never a valid initializer.

Predict first

Initializing all weights of a layer to 0 fails because:

Answer it in your own words, now, with nothing to choose from. The options are on the next slide — and picking the right one off a list is an easier skill than producing it.

Correct: every neuron gets identical gradients and never differentiates (broken symmetry)

Why: Identical weights → identical outputs → identical gradients → identical updates, forever. We verified dW1 has three identical rows, so the neurons never become distinct and the layer collapses to a single feature. Random init breaks the tie.

Check your understanding

Initializing all weights of a layer to 0 fails because:

• A. every neuron gets identical gradients and never differentiates (broken symmetry) (correct)
• B. the loss immediately becomes NaN
• C. the gradients explode
• D. it uses too much memory

Answer: A

Why: Identical weights → identical outputs → identical gradients → identical updates, forever. We verified dW1 has three identical rows, so the neurons never become distinct and the layer collapses to a single feature. Random init breaks the tie.

Why B tempts people

No NaN appears — the forward pass is a finite 0 and the loss is finite; learning is stuck, not numerically broken.

Why C tempts people

Zero weights make the gradient small and identical across neurons, not exploding — the failure is symmetry, not magnitude.

Why D tempts people

Memory is unaffected; the entire problem is the broken symmetry between neurons, not resource use.

Eliminate the wrong options

Gradient norm clipping rescales g to g·min(1, τ/‖g‖). Compared with clamping each component to [−τ, τ], it:

3 of these 4 are wrong. Strike them one at a time, and say what rules each one out before you strike the next. The survivor is the answer.

• A. preserves the gradient's direction; per-component clamping does not
• B. is identical to per-component clamping
• C. increases the gradient norm
• D. only works for 1-D gradients

Survives elimination: A

Why: Scaling the whole vector by one factor keeps every ratio between components fixed, so the direction is unchanged (only shorter). Per-component clamping shrinks big components more than small ones and rotates the vector — verified: [30,40]→[5,5] turns [0.6,0.8] into [0.707,0.707].

Check your understanding

Gradient norm clipping rescales g to g·min(1, τ/‖g‖). Compared with clamping each component to [−τ, τ], it:

• A. preserves the gradient's direction; per-component clamping does not (correct)
• B. is identical to per-component clamping
• C. increases the gradient norm
• D. only works for 1-D gradients

Answer: A

Why: Scaling the whole vector by one factor keeps every ratio between components fixed, so the direction is unchanged (only shorter). Per-component clamping shrinks big components more than small ones and rotates the vector — verified: [30,40]→[5,5] turns [0.6,0.8] into [0.707,0.707].

Why B tempts people

They differ: norm clipping keeps [0.6, 0.8]; value clamping bends it to [0.707, 0.707]. Only norm clipping preserves the descent direction.

Why C tempts people

It caps the norm DOWN to τ when ‖g‖ > τ (and leaves it alone otherwise); it never increases the norm.

Why D tempts people

It works in any dimension — scaling by a scalar is dimension-agnostic, which is precisely its advantage over per-component clamping.

Discussion prompt

Build rules: type every line yourself, apply ReLU each layer, seed the RNG so your numbers match, and read the final std — only He should stay O(1).

That is stated as though it always holds. Do one of two things: produce a case where it fails, or say precisely what rules such a case out. "It just does" is not on the menu.

Hint: Hunt at the extremes first — zero, one, negative, empty, equal. If every extreme survives, the reason they survive is the proof.

Step through it

Step through Milestone 2 — compare four initializations one row at a time. What is driving the change, and what would the row after the last one be?

1. Step 1: init is He
2. Step 2: init is Xavier
3. Step 3: init is too_big
4. Step 4: init is too_small

Comparison matrix

From Milestone 3 — clip a gradient: every row here is a choice with a cost. Fill the value (verified) column, then say which row you would actually pick and what you give up for it.

Comparison matrix

From The full program: refill the value (verified) column from what you know. The rest of the table is as it appeared.

Draw it

One page, no notation unless you need it: draw how these connect — The problem: depth multiplies · Clipping exploding gradients · Variance through a layer · Xavier initialization · He initialization · Backward & the 50-layer test. Put an arrow wherever one of them is what makes another possible, and label the arrow with why.