Lesson 15: Modern Optimizers

1. Modern Optimizers

Title

USAAIO · Lesson 15 · Week 5 (Optimization)

Why nobody trains with plain SGD anymore. We take one stretched bowl and race momentum, RMSprop, and Adam across it — each rule derived, hand-traced step by step, and built from scratch. Every loss number here came from real execution.

2. By the end of this lesson you can

Objectives

1. Say exactly why plain gradient descent crawls on an ill-conditioned bowl — its step is capped by the steep axis while the flat axis inches
2. Derive and hand-trace momentum v ← βv + g as a velocity that cancels oscillation and accumulates the consistent direction
3. Derive and hand-trace RMSprop s ← ρs + (1−ρ)g² as a per-parameter adaptive rate that rescales each axis
4. Assemble Adam = momentum + RMSprop + bias correction, and show at t=1 the raw step is 3.16× too big while the corrected step is exactly 1
5. Distinguish Adam vs AdamW (decoupled weight decay), explain warmup, and implement all three optimizers from scratch

3. What survived from Information Theory?

Warm-up

4. The problem: ill-conditioning

Section

Part 1 of 8

5. One bowl for the whole lesson

Concept

Every optimizer today races across the same quadratic bowl — a stretched paraboloid. We minimize:

\[ f(x_0, x_1) = \tfrac12\left(100\,x_0^2 + x_1^2\right) \]

Its gradient is one line, and we start every run from x = [1, 1], where the loss is ½(100 + 1) = 50.5.

\[ \nabla f(x) = \begin{bmatrix} 100\,x_0 \\ x_1 \end{bmatrix}, \qquad x^{(0)} = \begin{bmatrix}1\\1\end{bmatrix},\quad f(x^{(0)}) = 50.5 \]

6. Break it if you can: One bowl for the whole lesson

Counterexample

7. The Hessian and the condition number

Concept

The curvature of this bowl is its Hessian — the matrix of second derivatives. For our f it is diagonal and constant everywhere:

\[ H = \nabla^2 f = \begin{bmatrix} 100 & 0 \\ 0 & 1 \end{bmatrix} \]

condition number κ — The ratio of largest to smallest Hessian eigenvalue, κ = λ_max / λ_min. Here the eigenvalues are 100 and 1, so κ = 100. Big κ = a long, narrow valley — steep across one axis, nearly flat along the other.

κ = 100 is the entire villain of this lesson. Everything modern optimizers do is a response to large κ.

8. By analogy: The Hessian and the condition number

Analogy

9. A taco shell, not a soup bowl

Intuition

Picture the surface. Along x₀ the walls shoot up steeply (curvature 100); along x₁ the floor is nearly level (curvature 1). It's a long narrow taco shell, not a round soup bowl.

A ball dropped in rolls hard across the narrow direction and barely drifts down the long one. That mismatch — fast one way, slow the other — is what makes a single global step size fail.

10. Teach it back: A taco shell, not a soup bowl

Explain it

11. Plain gradient descent, and its speed limit

Concept

Plain gradient descent (GD) takes one global step size η down the negative gradient:

\[ x \leftarrow x - \eta\,\nabla f(x) \]

On a quadratic, GD is stable only while η < 2/L, where L is the largest curvature (the steep axis). Here L = 100, so:

\[ \eta < \frac{2}{L} = \frac{2}{100} = 0.02 \]

12. Say it in words: Plain gradient descent, and its speed limit

Translation

\( \eta < \frac{2}{L} = \frac{2}{100} = 0.02 \)

13. Guess the shape of the answer: Where the 2/L limit comes from

Estimation

14. Where the 2/L limit comes from

Worked example

On our separable bowl each axis is its own 1-D problem f = ½Lx² with ∇ = Lx. One GD step multiplies x by a fixed factor — derive it:

\[ x_{t+1} = x_t - \eta\,(L x_t) = (1 - \eta L)\,x_t \]

\[ |1 - \eta L| < 1 \;\Longleftrightarrow\; 0 < \eta < \frac{2}{L} \]

15. Fill in: L for Where the 2/L limit comes from

Comparison

16. What has to be given first: Watch GD crawl — full trace

Missing information

17. Watch GD crawl — full trace

Worked example

Run GD at η = 0.018 (just under the 0.02 ceiling). Because f is separable, each axis updates independently by a fixed factor: x₀ ← (1 − 100η)x₀ and x₁ ← (1 − η)x₁.

import numpy as np
def grad(x): return np.array([100*x[0], x[1]])
x, eta = np.array([1., 1.]), 0.018
for t in range(1, 11):
    x = x - eta*grad(x)
    loss = 0.5*(100*x[0]**2 + x[1]**2)
    print(t, np.round(x, 4), round(loss, 4))

18. What each one costs: Watch GD crawl — full trace

Trade off

19. The diagnosis: two failures at once

Concept

The trace shows both symptoms of ill-conditioning in one run:

1. x₀ zig-zags — the sign flips every step (−0.8, +0.64, −0.51…) because its update factor is negative. Energy wasted bouncing across the valley.
2. x₁ crawls — from 1.0 to 0.83 in ten steps. It needs a step 100× bigger, but η is pinned by x₀ and can't grow.

Momentum attacks the zig-zag; adaptive methods (RMSprop, Adam) attack the crawl. The next six parts build each fix and race them here.

20. Momentum

Section

Part 2 of 8 — a velocity

21. A heavy ball with mass

Intuition

Plain GD is a massless particle: it stops the instant the gradient does, and it turns on a dime every time the gradient flips. That's why it zig-zags.

Momentum gives the ball mass. Across the narrow valley the pushes alternate left–right and largely cancel; down the long axis the pushes all point the same way and accumulate into speed. The ball smooths the bounce and rolls straight down the valley.

22. The momentum update

Concept

Keep a velocity v — an exponential moving average of past gradients — and step along it instead of along the raw gradient:

\[ \begin{aligned} v &\leftarrow \beta\, v + \nabla f(x) \\ x &\leftarrow x - \eta\, v \end{aligned} \]

The friction/decay β (typically 0.9) says how much old velocity survives each step. β = 0 recovers plain GD; larger β gives the ball more inertia. We start v at the zero vector.

23. Why the velocity cancels and accumulates

Concept

Unroll the EMA: v is a decaying sum of every past gradient, v_t = Σ βᵏ g_{t−k}. On the steep axis the g's alternate sign, so the sum partly cancels — velocity stays small.

On the flat axis every g has the same sign, so the terms reinforce. A geometric series of same-sign pushes sums toward 1/(1−β) = 10× the single-step pull. The flat axis effectively gets a 10× boost — exactly where GD was crawling.

24. Predict the next row: Momentum, hand-traced (β=0.9, η=0.003)

Pattern

25. Momentum, hand-traced (β=0.9, η=0.003)

Worked example

Trace the velocity accumulating. Watch v[1] (flat axis) grow steadily from 1.0 while v[0] (steep axis) swings and averages toward zero.

import numpy as np
def grad(x): return np.array([100*x[0], x[1]])
x, v = np.array([1., 1.]), np.zeros(2)
for t in range(1, 9):
    g = grad(x)
    v = 0.9*v + g
    x = x - 0.003*v
    print(t, np.round(v, 3), np.round(x, 4))

26. Watch it run: Momentum, hand-traced (β=0.9, η=0.003)

Pattern

27. Momentum overshoots before it settles

Intuition

Notice the loss in that trace is not monotone: 25.0 → 2.9 → … → 21.1 → 22.7 → 0.43. A heavy ball builds speed on the steep axis and coasts past the bottom before inertia turns it around.

That's the price of mass: momentum can overshoot early, then converges fast once the transient settles. By step 80 it reaches loss 0.0008 — far past where GD stalled.

28. The two momentum conventions

Concept

You will see momentum written two ways, and they are not the same. Ours accumulates the raw gradient into v; the other (PyTorch's SGD(momentum=...)) blends with a (1−β) weight:

\[ \underbrace{v \leftarrow \beta v + g}_{\text{ours (raw sum)}} \qquad\text{vs}\qquad \underbrace{v \leftarrow \beta v + (1-\beta)\,g}_{\text{normalized average}} \]

The normalized form keeps v on the scale of a single gradient, so it needs an η about 1/(1−β) = 10× larger to match. Same trajectory, different η bookkeeping — know which one your library uses before you tune.

29. RMSprop

Section

Part 3 of 8 — per-axis rates

30. Why squared gradients, not absolute

Concept

RMSprop averages g², then divides by √s. Why the square-then-root instead of just averaging |g|? Because s is a running mean of squares, so √s is a root-mean-square — that's the RMS in the name.

Squaring is also what connects it to curvature: on a quadratic, the typical squared gradient on an axis scales with that axis's curvature, so √s estimates the local steepness and g/√s is a Newton-like, curvature-normalized step — without ever forming the Hessian.

31. RMSprop is a cheap second-order stand-in

Intuition

A true second-order method (Lesson 21) divides the step by the curvature itself — but that needs the Hessian, which is huge and expensive for real networks.

RMSprop fakes it: √s is a running, per-coordinate proxy for curvature built only from gradients you already computed. You get most of the axis-rescaling benefit of a second-order method at first-order cost. That trade is why adaptive methods dominate deep learning.

32. Give each axis its own step size

Intuition

Momentum still uses one global η. But the real disease is that the two axes want wildly different step sizes — the steep one small, the flat one large.

RMSprop's idea: measure how big each axis's gradients typically are, and divide each coordinate's step by that size. Loud axes get quieted; quiet axes get amplified. Every coordinate ends up moving at a comparable pace.

33. Where does each piece belong: Lesson 15: Modern Optimizers

Sorting

34. The RMSprop update

Concept

Track a running average of squared gradients, coordinate by coordinate, then divide the step by its square root:

\[ \begin{aligned} s &\leftarrow \rho\, s + (1-\rho)\, g^{\odot 2} \\ x &\leftarrow x - \eta\,\frac{g}{\sqrt{s} + \epsilon} \end{aligned} \]

Here g² and the division are elementwise (that's the ⊙). Defaults ρ = 0.9, ε = 1e-8 (a floor that stops division by zero). s starts at the zero vector.

√s is a per-coordinate estimate of gradient magnitude, so g/√s is roughly a unit-size step on every axis — the rescaling an ill-conditioned bowl needs.

35. Restore the missing line: RMSprop, hand-traced (ρ=0.9, η=0.05)

Fill the middle

36. RMSprop, hand-traced (ρ=0.9, η=0.05)

Worked example

At step 1 the raw gradient is [100, 1] — a 100:1 mismatch. Watch RMSprop erase it: after dividing by √s, both axes move by the identical amount.

import numpy as np
def grad(x): return np.array([100*x[0], x[1]])
x, s = np.array([1., 1.]), np.zeros(2)
for t in range(1, 6):
    g = grad(x)
    s = 0.9*s + 0.1*g*g
    step = g/(np.sqrt(s) + 1e-8)
    x = x - 0.05*step
    print(t, np.round(s, 3), np.round(step, 4), np.round(x, 5))

37. What happens as it grows: RMSprop, hand-traced (ρ=0.9, η=0.05)

Scale up

38. RMSprop's payoff

Concept

Because both axes descend in lockstep, the crawl is gone. By step 20 the loss is 0.25, by step 40 it is ≈ 1.5e-7, and by step 80 it has effectively hit machine zero — a completely different regime from GD's 0.03.

The catch: RMSprop rescales but has no memory of direction — it doesn't smooth the path. Combine its per-axis rescaling with momentum's smoothing and you get Adam.

39. Adam

Section

Part 4 of 8 — assemble the pieces

40. The two fixes are orthogonal

Intuition

Momentum fixes the direction of the step (smooth the zig-zag by averaging gradients over time). RMSprop fixes the scale of the step (rescale each axis by its own gradient size). These are independent knobs.

So there's nothing stopping you from turning both at once: average the gradient for direction, average the squared gradient for scale. That combination is Adam — and the only new subtlety is that both averages start biased at zero.

41. Adam = momentum + RMSprop

Concept

Adam keeps both running averages: a first moment m (momentum's velocity) and a second moment v (RMSprop's squared-gradient average).

\[ \begin{aligned} m &\leftarrow \beta_1\, m + (1-\beta_1)\, g \\ v &\leftarrow \beta_2\, v + (1-\beta_2)\, g^{\odot 2} \end{aligned} \]

Both start at the zero vector. Defaults β₁ = 0.9, β₂ = 0.999, ε = 1e-8. Note m uses the (1−β₁) weighting (a true average), unlike our bare momentum — a cosmetic difference that just rescales η.

42. The zero-init bias problem

Concept

Because m and v start at 0, early estimates are pulled toward zero. At t=1, with β₂ = 0.999:

\[ v_1 = 0.999\cdot 0 + 0.001\, g^2 = 0.001\,g^2 \]

That is 1000× too small as an estimate of g²! Its square root, √v ≈ 0.0316\,|g|, is 31.6× too small — so dividing by it would make the step 31.6× too large. The estimate hasn't warmed up yet.

43. Finish it with less help: Bias correction, dissected at t=1

Faded example

44. Bias correction, dissected at t=1

Worked example

The fix: divide each moment by (1 − βᵗ) to undo the shrink. Watch what it does to the actual step magnitude on the steep axis at t=1, where g₀ = 100:

import numpy as np
g = 100.0                      # steep-axis gradient at t=1
m, v = 0.1*g, 0.001*g*g        # m=(1-b1)g, v=(1-b2)g^2
raw = m/(np.sqrt(v) + 1e-8)               # NO correction
mh, vh = m/(1-0.9**1), v/(1-0.999**1)     # bias-corrected
cor = mh/(np.sqrt(vh) + 1e-8)
print('raw step dir:', round(raw, 4))
print('corrected   :', round(cor, 4))

45. Which is which, by raw (no BC)

Discrimination

46. The full Adam update

Concept

Correct both moments with their own factor, then take the RMSprop-style step:

\[ \hat m = \frac{m}{1-\beta_1^{\,t}}, \qquad \hat v = \frac{v}{1-\beta_2^{\,t}}, \qquad x \leftarrow x - \eta\,\frac{\hat m}{\sqrt{\hat v} + \epsilon} \]

As t grows, βᵗ → 0, so (1 − βᵗ) → 1 and the correction quietly fades away — it only matters in the first dozen-or-so steps. Adam is the default optimizer for nearly all deep learning.

47. Adam's step size is self-bounding

Intuition

Here's why Adam is so forgiving to tune. The update per coordinate is η·m̂/√v̂, and both m̂ and √v̂ are averages of the same gradients — so their ratio is roughly sign(g), of magnitude near 1.

That means each parameter moves by about ±η regardless of its gradient's scale. A coordinate with a gradient of 100 and one with a gradient of 0.001 both step by ~η. The raw gradient magnitude is divided out — which is exactly the cure for a 100:1 bowl, and why one η now works for all axes.

48. Adam, hand-traced (η=0.1)

Worked example

Full Adam on the bowl. The bias-corrected m̂[0] starts at exactly 100 (the true gradient), and the step marches both axes down together — no zig-zag, no crawl.

import numpy as np
def grad(x): return np.array([100*x[0], x[1]])
x, m, v = np.array([1., 1.]), np.zeros(2), np.zeros(2)
for t in range(1, 11):
    g = grad(x)
    m = 0.9*m + 0.1*g
    v = 0.999*v + 0.001*g*g
    mh, vh = m/(1-0.9**t), v/(1-0.999**t)
    x = x - 0.1*mh/(np.sqrt(vh) + 1e-8)
    print(t, np.round(x, 5), round(0.5*(100*x[0]**2+x[1]**2), 4))

49. Watch it run: Adam, hand-traced (η=0.1)

Pattern

50. Something is wrong here: skipping Adam's bias correction

Anomaly

51. Trap: skipping Adam's bias correction

Trap

52. Break it on purpose: skipping Adam's bias correction

Break the constraint

53. Restore the missing line: Bias correction on vs off — head to head

Fill the middle

54. Bias correction on vs off — head to head

Worked example

One flag toggles the /(1−βᵗ) terms. Run both variants on the bowl at η = 0.1 and compare the loss at three checkpoints.

import numpy as np
def grad(x): return np.array([100*x[0], x[1]])
def loss(x): return 0.5*(100*x[0]**2 + x[1]**2)
def adam(bc, steps=30, lr=0.1, b1=0.9, b2=0.999, eps=1e-8):
    x, m, v = np.array([1.,1.]), np.zeros(2), np.zeros(2)
    hist = {}
    for t in range(1, steps+1):
        g = grad(x)
        m = b1*m + (1-b1)*g
        v = b2*v + (1-b2)*g*g
        if bc: mh, vh = m/(1-b1**t), v/(1-b2**t)
        else:  mh, vh = m, v
        x = x - lr*mh/(np.sqrt(vh)+eps)
        hist[t] = loss(x)
    return hist
on, off = adam(True), adam(False)
print('with BC :', [round(on[k], 4)  for k in (1, 5, 30)])
print('no BC   :', [round(off[k], 4) for k in (1, 5, 30)])

55. Work backwards from the answer: Bias correction on vs off — head to head

Reverse engineer

56. Race the four

Section

Part 5 of 8 — same bowl, same start

57. Guess the shape of the answer: All four optimizers, one bowl

Estimation

58. All four optimizers, one bowl

Worked example

Same problem, same start [1,1], start loss 50.5. Each optimizer uses its own tuned η. This block runs GD, momentum, RMSprop, and Adam and prints the loss at four checkpoints.

import numpy as np
def grad(x): return np.array([100*x[0], x[1]])
def loss(x): return 0.5*(100*x[0]**2 + x[1]**2)

def adam(steps, lr=0.1, b1=0.9, b2=0.999, eps=1e-8):
    x, m, v = np.array([1.,1.]), np.zeros(2), np.zeros(2)
    hist = {}
    for t in range(1, steps+1):
        g = grad(x)
        m = b1*m + (1-b1)*g
        v = b2*v + (1-b2)*g*g
        mh, vh = m/(1-b1**t), v/(1-b2**t)
        x = x - lr*mh/(np.sqrt(vh)+eps)
        hist[t] = loss(x)
    return hist

h = adam(80)
print([round(h[k], 4) for k in (10, 20, 40, 80)])

59. Reading the race honestly

Concept

The table isn't a clean 'Adam wins' story, and that's the point. Momentum and Adam both overshoot mid-run (momentum 15.5 at step 10; Adam 3.7 at step 20) as their velocity coasts past the bottom, then recover.

RMSprop, with no momentum to overshoot, descends smoothly straight to machine zero on this pure quadratic. Every method beats GD's 0.027 floor by orders of magnitude — because every method frees the flat axis.

60. The shape of each trajectory

Concept

The loss curve reveals each method's personality on the same bowl:

• GD — a monotone but painfully slow decay; the flat axis never accelerates, so it flatlines near 0.03.
• Momentum — a big early hump (loss up to 15.5) as the ball overshoots, then a fast plunge to 0.0008.
• RMSprop — the smoothest: no hump, straight down to machine zero.
• Adam — a small overshoot (0.29 → 3.7) from its momentum half, then it settles like RMSprop.

The lesson: momentum buys speed at the cost of an early overshoot; adaptivity buys smoothness. Adam takes both, so it inherits a little of both traits.

61. Something is wrong here: is Adam always the best choice?

Anomaly

62. Trap: is Adam always the best choice?

Trap

63. Which of these survive contact with Lesson 15: Modern Optimizers?

Two truths and a lie

64. AdamW & warmup

Section

Part 6 of 8 — the modern defaults

65. Weight decay, and where Adam puts it

Concept

Weight decay shrinks each weight toward 0 every step — the L2 regularizer of Lesson 12. The classic way folds it into the gradient as g + λθ, so it flows through Adam's adaptive √v̂ divisor.

\[ \text{Adam+L2:}\quad g \leftarrow \nabla f + \lambda\theta, \qquad x \leftarrow x - \eta\,\frac{\hat m}{\sqrt{\hat v}+\epsilon} \]

The problem: dividing the decay by √v̂ means weights on loud (large-gradient) axes decay less than weights on quiet ones. The regularization strength gets distorted per-coordinate — not what you asked for.

66. AdamW: decouple the decay

Concept

AdamW applies the decay as its own term, outside the adaptive step, so every weight decays by the same fraction η·λ:

\[ x \leftarrow x - \eta\,\frac{\hat m}{\sqrt{\hat v}+\epsilon} - \eta\,\lambda\,\theta \]

The decay η·λ·θ no longer passes through √v̂, so it is decoupled from the gradient geometry. This is the standard recipe for training transformers.

67. Finish it with less help: AdamW vs Adam+L2 differ on step one

Faded example

68. AdamW vs Adam+L2 differ on step one

Worked example

One step from θ = [1, 1] with λ = 0.1, η = 0.1. The two recipes land on different points — proof the placement matters.

import numpy as np
def grad(x): return np.array([100*x[0], x[1]])
b1,b2,eta,eps,lam = 0.9,0.999,0.1,1e-8,0.1
# Adam + L2: decay folded into the gradient
x = np.array([1.,1.]); g = grad(x) + lam*x
m, v = (1-b1)*g, (1-b2)*g*g
mh, vh = m/(1-b1), v/(1-b2)
x_l2 = x - eta*mh/(np.sqrt(vh)+eps)
# AdamW: decay applied separately
x = np.array([1.,1.]); g = grad(x)
m, v = (1-b1)*g, (1-b2)*g*g
mh, vh = m/(1-b1), v/(1-b2)
x_w = x - eta*mh/(np.sqrt(vh)+eps) - eta*lam*x
print('Adam+L2:', np.round(x_l2,4), ' AdamW:', np.round(x_w,4))

69. Fill in: x₀ after 1 step for AdamW vs Adam+L2 differ on step one

Comparison

70. Learning-rate warmup

Concept

We saw the flip side of bias correction: very early, the second moment v̂ is built from just a few gradients and is noisy. Even corrected, a big η on a shaky estimate can destabilize training.

Warmup ramps η linearly from ≈0 over the first few hundred steps, then cosine-anneals it back down toward 0. The ramp buys time for v̂ to settle before full-size steps hit; transformers essentially require it.

71. Teach it back: Learning-rate warmup

Explain it

72. A warmup + cosine schedule

Worked example

Linear ramp over the first W steps to base, then a cosine decay to 0 across the remaining T − W. This block prints the η the optimizer would use at several steps.

import math
def lr_sched(t, base=0.1, W=10, T=80):
    if t < W:
        return base * (t+1)/W          # linear warmup
    prog = (t - W)/(T - W)
    return base * 0.5*(1 + math.cos(math.pi*prog))   # cosine decay
for t in [0, 1, 4, 9, 20, 40, 79]:
    print(t, round(lr_sched(t), 5))

73. Work backwards from the answer: A warmup + cosine schedule

Reverse engineer

74. Warmup and bias correction solve the same era, differently

Intuition

Both target the fragile first few steps, where v̂ is estimated from almost no data. Bias correction fixes the systematic shrink (the 1000× underestimate at t=1); warmup handles the random noise that correction can't remove.

Correction makes the early step the right size on average; warmup keeps it small so a single unlucky, noisy v̂ can't send the weights flying before the averages settle. On a huge transformer, one bad early step can poison the whole run — so you use both.

75. The toolkit

Section

Part 7 of 8 — pattern & checks

76. Rebuild the recipe: The optimizer toolkit

Ranking

77. The optimizer toolkit

Pattern

1. Momentum — v = βv + g; x -= ηv. A velocity EMA: cancels oscillation, accumulates the consistent direction. β ≈ 0.9.
2. RMSprop — s = ρs + (1−ρ)g²; x -= ηg/(√s+ε). Per-parameter rate: divide each axis by its own gradient size.
3. Adam — both moments, then bias-correct m̂ = m/(1−β₁ᵗ), v̂ = v/(1−β₂ᵗ), then x -= η m̂/(√v̂+ε).
4. AdamW — decouple weight decay: x -= η m̂/(√v̂+ε) − ηλθ (decay outside the adaptive divisor).
5. Schedule — warmup η up, then cosine-decay it down; near-mandatory for transformers.

78. Where does it stop working: The optimizer toolkit

Edge cases

79. Rule out three: Check yourself — why GD crawls

Elimination

80. Check yourself — why GD crawls

Check

Think about what caps the step size on our bowl.

81. Answer it before you see the options: Check yourself — Adam's parts

Prediction

82. Check yourself — Adam's parts

Check

What two ideas is Adam built from?

83. Answer it before you see the options: Check yourself — bias correction

Prediction

84. Check yourself — bias correction

Check

Recall what happens at t=1 with β₂ = 0.999.

85. Rule out three: Check yourself — AdamW

Elimination

86. Check yourself — AdamW

Check

Where does AdamW put the weight decay?

87. Your turn: build the optimizers

Section

Part 8 of 8 — the project

88. Project: momentum, RMSprop, Adam from scratch

Concept

Implement all three update rules yourself and run each on the same ill-conditioned bowl f(x) = ½(100x₀² + x₁²) from [1, 1]. You've traced every rule by hand — now assemble them in code.

Build rules: type every line yourself, initialize all state to zeros, and for Adam loop t from 1 and never drop the /(1−βᵗ) correction. Run after each milestone and read the loss.

89. By analogy: Project: momentum, RMSprop, Adam from scratch

Analogy

90. Restore the missing line: Milestone 1 — momentum

Fill the middle

91. Milestone 1 — momentum

Worked example

Your turn: implement momentum and run 80 steps at η = 0.003. Predict aloud whether it beats GD's step-80 loss of 0.027 before you print.

Hint: keep v = np.zeros(2); each step g = grad(x); v = 0.9*v + g; x = x - lr*v. Expect an early overshoot in the loss, then a fast finish.

import numpy as np
def grad(x): return np.array([100*x[0], x[1]])
x, v, lr = np.array([1., 1.]), np.zeros(2), 0.003
for t in range(1, 81):
    g = grad(x)
    v = 0.9*v + g
    x = x - lr*v
    if t in (10, 40, 80):
        print(t, round(0.5*(100*x[0]**2 + x[1]**2), 4))

92. Watch it run: Milestone 1 — momentum

Pattern

93. Predict the next row: Milestone 2 — RMSprop

Pattern

94. Milestone 2 — RMSprop

Worked example

Your turn: implement RMSprop at η = 0.05. Predict which axis its √s division speeds up most before you run.

Hint: s = np.zeros(2); each step s = 0.9*s + 0.1*g*g; x = x - lr*g/(np.sqrt(s) + 1e-8). On step 1 both axes should move by the identical amount.

import numpy as np
def grad(x): return np.array([100*x[0], x[1]])
x, s, lr = np.array([1., 1.]), np.zeros(2), 0.05
for t in range(1, 81):
    g = grad(x)
    s = 0.9*s + 0.1*g*g
    x = x - lr*g/(np.sqrt(s) + 1e-8)
    if t in (20, 40, 80):
        print(t, round(0.5*(100*x[0]**2 + x[1]**2), 8))

95. What happens as it grows: Milestone 2 — RMSprop

Scale up

96. What has to be given first: Milestone 3 — Adam

Missing information

97. Milestone 3 — Adam

Worked example

Your turn: implement Adam with bias correction at η = 0.1. Predict what breaks if you delete the mh, vh lines and step with raw m, v instead.

Hint: track m, v = np.zeros(2), np.zeros(2); each step compute mh = m/(1-0.9**t), vh = v/(1-0.999**t); then x -= lr*mh/(np.sqrt(vh)+1e-8). Loop t from 1.

import numpy as np
def grad(x): return np.array([100*x[0], x[1]])
x, m, v, lr = np.array([1., 1.]), np.zeros(2), np.zeros(2), 0.1
for t in range(1, 81):
    g = grad(x)
    m = 0.9*m + 0.1*g
    v = 0.999*v + 0.001*g*g
    mh, vh = m/(1-0.9**t), v/(1-0.999**t)
    x = x - lr*mh/(np.sqrt(vh) + 1e-8)
    if t in (10, 20, 80):
        print(t, round(0.5*(100*x[0]**2 + x[1]**2), 5))

98. Watch it run: Milestone 3 — Adam

Pattern

99. Guess the shape of the answer: Milestone 4 — race your three vs GD

Estimation

100. Milestone 4 — race your three vs GD

Worked example

Your turn: wrap each rule in a function returning the final loss, and print all four side by side. Predict the ranking at step 80 before you run.

Hint: each function starts x=np.array([1.,1.]) and its own zero state, loops 80 steps, returns loss(x). Reuse grad and loss helpers.

import numpy as np
def grad(x): return np.array([100*x[0], x[1]])
def loss(x): return 0.5*(100*x[0]**2 + x[1]**2)
def run(kind, lr, steps=80):
    x, v, s, m, w = np.array([1.,1.]), np.zeros(2), np.zeros(2), np.zeros(2), np.zeros(2)
    for t in range(1, steps+1):
        g = grad(x)
        if kind == 'gd':   x = x - lr*g
        elif kind == 'mom': v = 0.9*v + g;            x = x - lr*v
        elif kind == 'rms': s = 0.9*s + 0.1*g*g;      x = x - lr*g/(np.sqrt(s)+1e-8)
        elif kind == 'adam':
            m = 0.9*m + 0.1*g; w = 0.999*w + 0.001*g*g
            mh, vh = m/(1-0.9**t), w/(1-0.999**t)
            x = x - lr*mh/(np.sqrt(vh)+1e-8)
    return round(loss(x), 5)
for k, lr in [('gd',0.018),('mom',0.003),('rms',0.05),('adam',0.1)]:
    print(k, run(k, lr))

101. What each one costs: Milestone 4 — race your three vs GD

Trade off

102. The full program

Concept

import numpy as np
def grad(x): return np.array([100*x[0], x[1]])
def loss(x): return 0.5*(100*x[0]**2 + x[1]**2)

def adam(steps=80, lr=0.1, b1=0.9, b2=0.999, eps=1e-8):
    x, m, v = np.array([1., 1.]), np.zeros(2), np.zeros(2)
    for t in range(1, steps+1):
        g = grad(x)
        m = b1*m + (1-b1)*g              # momentum piece
        v = b2*v + (1-b2)*g*g            # RMSprop piece
        mh, vh = m/(1-b1**t), v/(1-b2**t)  # bias correction
        x = x - lr*mh/(np.sqrt(vh)+eps)
    return loss(x)

print(round(adam(), 5))     # 0.00147

If your Adam drives the loss from 50.5 to about 0.0015 while plain GD is still stuck near 0.027 — you've built the optimizer that trains modern networks.

103. Fill in: value for The full program

Comparison

104. Show it off

Concept

Slides closed, out loud: explain (1) why one global η crawls on a κ = 100 bowl, (2) what RMSprop's √s division does to the steep axis, and (3) why bias correction matters most at step 1.

Stretch (homework): add a warmup + cosine schedule to your Adam and re-race; then swap in AdamW's decoupled decay. These optimizers reappear in the full training loop (Week 14) and transformer training (Week 29).

105. Break it if you can: Show it off

Counterexample

106. Connect it up: Lesson 15: Modern Optimizers

Connect it up

107. What you can do now

Recap

• Explain ill-conditioning: one global η is capped by the steep axis (η < 2/L), so the flat axis crawls — the whole reason for modern optimizers
• Derive and trace momentum (velocity EMA that cancels zig-zag) and RMSprop (per-axis g/√s rescaling)
• Assemble Adam = momentum + RMSprop + bias correction, and show the raw step is 3.16× too big at t=1 while the corrected step is exactly 1
• Distinguish AdamW (decoupled −ηλθ decay) from Adam+L2, and explain why warmup stabilizes the noisy early v̂
• Implement all three from scratch and race them: every method beats GD's 0.027 floor on our bowl