Lesson 62: Learning Rate Schedules

Discussion prompt

Before we open Lesson 62: Learning Rate Schedules: without looking back, what was the main idea of Multiclass Classification — Softmax, OvR, OvO, and what could you do by the end of it that you could not do before?

Hint: One sentence for the idea, one for the skill. If the second one is blank, that is the part to revisit.

Comparison matrix

From Fixed LR leaves a trade-off unresolved: refill the early training column from what you know. The rest of the table is as it appeared.

Comparison matrix

From Warmup: ramp from 0 to target_lr: every row here is a choice with a cost. Fill the lr (target = 1e-3, N=100) column, then say which row you would actually pick and what you give up for it.

Predict first

Implement warmup with LambdaLR — the lambda returns a multiplier on base_lr.

Commit before you compute: what does Warmup via LambdaLR come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: lr at step 100 = 1.0000e-03 — LR reached target

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. The lambda returns 1.0 at step=N_warm, so the multiplier is exactly 1.0 and the full base_lr is active.

Step through it

Step through Warmup via LambdaLR one row at a time. What is driving the change, and what would the row after the last one be?

1. Step 1: step is 0
2. Step 2: step is 25
3. Step 3: step is 50
4. Step 4: step is 100

Step through it

Step through Cosine annealing — smooth decay one row at a time. What is driving the change, and what would the row after the last one be?

1. Step 1: t (of T=100) is 0
2. Step 2: t (of T=100) is 25
3. Step 3: t (of T=100) is 50
4. Step 4: t (of T=100) is 75
5. Step 5: t (of T=100) is 100

Discussion prompt

PyTorch's built-in CosineAnnealingLR matches the formula exactly with T_max=T, eta_min=LR_min.

What do you need to know — or decide — before the first line can be written? List everything the problem has to hand you.

Hint: Anything you would have to invent to get started is a thing the problem must supply.

Step through it

Step through CosineAnnealingLR in PyTorch and watch the numbers move. Now imagine the input ten times bigger: which column is the one that stops this being practical?

1. Step 1: t is 0
2. Step 2: t is 25
3. Step 3: t is 50
4. Step 4: t is 75
5. Step 5: t is 100

Discussion prompt

Add a restart every T_0 steps: after reaching LR_min, snap back to LR_max. Each restart helps the optimizer escape a sharp local minimum (Lesson 50's loss-surface intuition).

That is stated as though it always holds. Do one of two things: produce a case where it fails, or say precisely what rules such a case out. "It just does" is not on the menu.

Hint: Hunt at the extremes first — zero, one, negative, empty, equal. If every extreme survives, the reason they survive is the proof.

Predict first

A student writes this, and it looks reasonable:

The scheduler is called at the top of the loop, before optimizer.step().

It is wrong. Say what breaks — and say it before you turn the page.

Correct: PyTorch raises a warning: 'Detected call of lr_scheduler.step() before optimizer.step().' The scheduler skips the first LR value, so step 0 uses step 1's LR and the whole trace shifts by one.

Call scheduler.step() AFTER optimizer.step(), at the end of each iteration.

Why: PyTorch raises a warning: 'Detected call of lr_scheduler.step() before optimizer.step().' The scheduler skips the first LR value, so step 0 uses step 1's LR and the whole trace shifts by one.

Discussion prompt

The rule this trap just fixed:

The optimizer applies the current LR first, then the scheduler advances to the next LR. Order matters — PyTorch enforces it with a warning in 1.1+ and will eventually error.

Now break it on purpose. Build a case that violates it and follow the consequences until something visibly fails. Where does the failure first show up — and would you have noticed it if you had not been looking?

Hint: The dangerous rules are the ones whose violation still produces an answer. If yours fails loudly, try to find one that fails quietly.

Discussion prompt

Explain One-cycle: LR up then down, momentum inverse by analogy to something with no Machine Learning in it at all — a queue, a recipe, a map, a bank balance, whatever fits. Then say where your analogy breaks.

Hint: An analogy that never breaks is not an analogy, it is the same idea wearing a hat. Find the seam — that is the part that is actually new.

Predict first

Verify the LR/momentum anti-correlation with PyTorch OneCycleLR on a 100-step schedule, max_lr=0.1, pct_start=0.3.

Commit before you compute: what does OneCycleLR — LR and momentum trace come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: step 0: lr=4.28e-3, mom=0.9497 → step 30: lr=9.98e-2, mom=0.8502 → step 99: lr=5.07e-5, mom=0.9499

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. LR peaks at step 30 (= 100 × 0.3) while momentum hits its minimum — confirming the inverse relationship Smith described.

Comparison matrix

From OneCycleLR — LR and momentum trace: refill the momentum column from what you know. The rest of the table is as it appeared.

Discussion prompt

Explain Inverse square root — the Transformer schedule by analogy to something with no Machine Learning in it at all — a queue, a recipe, a map, a bank balance, whatever fits. Then say where your analogy breaks.

Hint: An analogy that never breaks is not an analogy, it is the same idea wearing a hat. Find the seam — that is the part that is actually new.

Predict first

Implement the exact Vaswani formula in LambdaLR — one function, both warmup and decay.

Commit before you compute: what does Transformer schedule from scratch with LambdaLR come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: Peak at step=4000: lr=6.99e-04 = d_model^-0.5 × N_warm^-0.5 = 512^-0.5 × 4000^-0.5

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. The two terms in the min() cross exactly at t = N_warm.

Step through it

Step through Transformer schedule from scratch with LambdaLR one row at a time. What is driving the change, and what would the row after the last one be?

1. Step 1: step is 1
2. Step 2: step is 1000
3. Step 3: step is 4000
4. Step 4: step is 8000
5. Step 5: step is 16000

Predict first

A student writes this, and it looks reasonable:

Set base_lr=1e-3 in Adam, then define the lambda to return the full formula value.

It is wrong. Say what breaks — and say it before you turn the page.

Correct: LambdaLR multiplies base_lr by the lambda.

Set base_lr=1.0 in Adam so the lambda IS the effective LR.

Why: LambdaLR multiplies base_lr by the lambda. If base_lr is not 1.0, the effective LR is base_lr × lambda — it will be 1000× too small (or too large) compared to the intended schedule.

Sort into buckets

Some of these hold up and some are the exact mistakes this lesson is built to prevent. Sort them.

Holds up

A large LR covers ground fast early, but overshoots sharp valleys late. A small LR lands precisely but wastes most of the budget crawling.; Adam tracks a per-parameter variance estimate v. At step 0 that estimate is 0 — the effective step size is garbage. Warmup lets v stabilize before the full LR kicks in.; Decay LR along a cosine curve: starts at LR_max, ends at LR_min. Smooth means the gradient never sees a sudden LR cliff.

Breaks

The scheduler is called at the top of the loop, before optimizer.step().; Set base_lr=1e-3 in Adam, then define the lambda to return the full formula value.

sound

These are stated as this lesson states them — each one survives the edge cases Lesson 62: Learning Rate Schedules puts it through.

flawed

Each of these is lifted from a trap in this deck: reasonable-sounding, and wrong in a way that only shows up once you rely on it.

Put in order

These are the steps of LR schedule selection recipe, scrambled. Put them back in order before the next slide shows you.

1. Always warmup Adam — N_warm ≈ 5–10 % of total steps; use LambdaLR
2. Cosine annealing — default decay choice; smooth, no hyperparameter tuning
3. SGDR — add CosineAnnealingWarmRestarts when plateau is a risk (non-convex losses)
4. One-cycle — for fast single-run training; set max_lr via LR-finder, pct_start=0.3
5. Transformer schedule — for self-attention models; set base_lr=1.0, tune d_model & N_warm
6. Implement any schedule with LambdaLR — the lambda is just a function of step

Why: This is the order the recipe itself gives. Recalling the sequence without the slide in front of you is the difference between recognising the method and being able to run it — most of what goes wrong in practice is a step done out of turn.

Discussion prompt

LR schedule selection recipe works on the cases you have just seen. Push it to the edge: what is the most degenerate input it still handles — empty, zero, one item, everything equal — and what is the first case where it stops being true? Name the case, not just "it breaks".

Hint: Try the smallest legal input, then the largest, then the one where two things collide. Methods are specified at their edges; the middle takes care of itself.

Eliminate the wrong options

Why does linear warmup stabilize Adam in early training?

3 of these 4 are wrong. Strike them one at a time, and say what rules each one out before you strike the next. The survivor is the answer.

• A. Adam's variance estimate v starts at 0; a small LR lets v accumulate before the full step size is trusted
• B. A small LR reduces gradient noise by averaging more mini-batches
• C. Warmup prevents the loss from overshooting in cosine annealing
• D. Adam's bias correction makes early steps too small without warmup

Survives elimination: A

Why: Adam's per-parameter second moment estimate v(0) = 0. The effective step size is gradient / sqrt(v + ε) — when v ≈ 0, ε dominates and the step is large and unstable. Warmup keeps LR tiny until v is meaningful.

Check your understanding

Why does linear warmup stabilize Adam in early training?

• A. Adam's variance estimate v starts at 0; a small LR lets v accumulate before the full step size is trusted (correct)
• B. A small LR reduces gradient noise by averaging more mini-batches
• C. Warmup prevents the loss from overshooting in cosine annealing
• D. Adam's bias correction makes early steps too small without warmup

Answer: A

Why: Adam's per-parameter second moment estimate v(0) = 0. The effective step size is gradient / sqrt(v + ε) — when v ≈ 0, ε dominates and the step is large and unstable. Warmup keeps LR tiny until v is meaningful.

Why B tempts people

Gradient noise is reduced by a larger batch, not a smaller LR. Mini-batch size is unchanged during warmup.

Why C tempts people

Warmup is independent of cosine annealing — it's a concern about Adam's internal state, not about LR decay.

Why D tempts people

Adam's bias correction (×1/(1-β^t)) actually inflates early steps, not deflates them — warmup counters this inflation.

Predict first

What does SGDR add over plain cosine annealing?

Answer it in your own words, now, with nothing to choose from. The options are on the next slide — and picking the right one off a list is an easier skill than producing it.

Correct: Periodic restarts back to LR_max, helping escape sharp local minima

Why: SGDR (Loshchilov & Hutter 2017) appends a snap-back to LR_max at the end of each T_0-step cycle. This lets the optimizer escape narrow minima it would otherwise converge into — the restart provides the perturbation.

Check your understanding

What does SGDR add over plain cosine annealing?

• A. Periodic restarts back to LR_max, helping escape sharp local minima (correct)
• B. An inverse-cosine momentum schedule coupled to the LR
• C. A warmup phase at the start of each cosine period
• D. A linear final decay after the cosine reaches LR_min

Answer: A

Why: SGDR (Loshchilov & Hutter 2017) appends a snap-back to LR_max at the end of each T_0-step cycle. This lets the optimizer escape narrow minima it would otherwise converge into — the restart provides the perturbation.

Why B tempts people

Momentum coupling is the one-cycle policy (Smith 2019), not SGDR.

Why C tempts people

SGDR does not add a warmup within each cosine period — the restart jumps directly to LR_max.

Why D tempts people

SGDR does not append a linear decay; it snaps back to LR_max and starts a new cosine cycle.

Eliminate the wrong options

For d_model=512 and N_warm=4000, at which step does the Transformer LR reach its peak, and what is that peak?

3 of these 4 are wrong. Strike them one at a time, and say what rules each one out before you strike the next. The survivor is the answer.

• A. step 4000, lr ≈ 6.99e-04
• B. step 512, lr ≈ 1.0e-03
• C. step 1, lr ≈ 1.75e-04
• D. step 8000, lr ≈ 4.94e-04

Survives elimination: A

Why: The two terms cross when t^-0.5 = t × N_warm^-1.5, which gives t = N_warm = 4000. Peak = d_model^-0.5 × N_warm^-0.5 = 512^-0.5 × 4000^-0.5 ≈ 6.99e-04. Verified in Python.

Check your understanding

For d_model=512 and N_warm=4000, at which step does the Transformer LR reach its peak, and what is that peak?

• A. step 4000, lr ≈ 6.99e-04 (correct)
• B. step 512, lr ≈ 1.0e-03
• C. step 1, lr ≈ 1.75e-04
• D. step 8000, lr ≈ 4.94e-04

Answer: A

Why: The two terms cross when t^-0.5 = t × N_warm^-1.5, which gives t = N_warm = 4000. Peak = d_model^-0.5 × N_warm^-0.5 = 512^-0.5 × 4000^-0.5 ≈ 6.99e-04. Verified in Python.

Why B tempts people

d_model appears as a scale factor (d_model^-0.5), not as a step index — the peak is always at step=N_warm.

Why C tempts people

Step 1 is the first warmup step, where lr ≈ 1.70e-07 (nearly zero), far from the peak.

Why D tempts people

Step 8000 is past the peak — by then the sqrt-decay dominates and lr ≈ 4.94e-04, lower than the peak.

Discussion prompt

Implement all five schedules from scratch using LambdaLR. Each lambda is a pure Python function — no scheduler classes except LambdaLR itself.

That is stated as though it always holds. Do one of two things: produce a case where it fails, or say precisely what rules such a case out. "It just does" is not on the menu.

Hint: Hunt at the extremes first — zero, one, negative, empty, equal. If every extreme survives, the reason they survive is the proof.

Step through it

Step through Milestone 1 — warmup + cosine lambdas one row at a time. What is driving the change, and what would the row after the last one be?

1. Step 1: step is 0
2. Step 2: step is 25
3. Step 3: step is 50
4. Step 4: step is 75
5. Step 5: step is 100

Comparison matrix

From Milestone 2 — SGDR lambda: every row here is a choice with a cost. Fill the multiplier (= lr if base=1) column, then say which row you would actually pick and what you give up for it.

Comparison matrix

From The full program: all five schedules: refill the step 100 column from what you know. The rest of the table is as it appeared.

Draw it

One page, no notation unless you need it: draw how these connect — Why schedules? The fixed-LR problem · Schedule 1: Linear Warmup · Schedule 2: Cosine Annealing · Schedule 3: One-Cycle Policy · Schedule 4: Transformer / Inverse Sqrt · Your turn: implement all five schedules. Put an arrow wherever one of them is what makes another possible, and label the arrow with why.