Lesson 11: Convex Optimization

Discussion prompt

Before we open Lesson 11: Convex Optimization: without looking back, what was the main idea of Object Detection & YOLO, and what could you do by the end of it that you could not do before?

Hint: One sentence for the idea, one for the skill. If the second one is blank, that is the part to revisit.

Discussion prompt

We carry one function through the whole lesson so every slide compounds. It is a 2-variable quadratic in w = (w₁, w₂):

That is stated as though it always holds. Do one of two things: produce a case where it fails, or say precisely what rules such a case out. "It just does" is not on the menu.

Hint: Hunt at the extremes first — zero, one, negative, empty, equal. If every extreme survives, the reason they survive is the proof.

Discussion prompt

Explain Convex = one bowl, no dents by analogy to something with no Machine Learning in it at all — a queue, a recipe, a map, a bank balance, whatever fits. Then say where your analogy breaks.

Hint: An analogy that never breaks is not an analogy, it is the same idea wearing a hat. Find the seam — that is the part that is actually new.

Predict first

Claim: f(w)=½(w₁²+4w₂²) is convex. Let's not take it on faith — pick x=(4,1) and y=(−2,3), sweep α, and check the chord inequality numerically. Runnable as written:

Commit before you compute: what does Verify the chord inequality on our bowl come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: At every α, f(mix) ≤ chord — the inequality holds

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. The surface height never exceeds the chord height.

Discussion prompt

Explain The picture: surface below its chord to a student a year behind you. No notation, no jargon they have not met — and it still has to be true.

Hint: If your explanation needs a symbol they have never seen, you are describing the notation rather than the idea.

Discussion prompt

Test the first-order condition on our bowl. Anchor the tangent at x=(2,1), where f(x)=4 and ∇f(x)=[2,4], then check f(y) ≥ tangent(y) at several y. Runnable:

What do you need to know — or decide — before the first line can be written? List everything the problem has to hand you.

Hint: Anything you would have to invent to get started is a thing the problem must supply.

Sort into buckets

Sort these by f(y) ≥ tangent?, from memory, without looking back at Verify the tangent lies below. Telling them apart on the spot is the skill; the table is only where the answer happens to be written down.

True

[0, 0]; [3, 2]; [−1, 1]

True (touches at y=x)

[2, 1]

g1

f(y) ≥ tangent? is "True" for [0, 0], [3, 2], [−1, 1] — that is what the table on "Verify the tangent lies below" records, and it is the single property separating this group from the rest.

g2

f(y) ≥ tangent? is "True (touches at y=x)" for [2, 1] — that is what the table on "Verify the tangent lies below" records, and it is the single property separating this group from the rest.

Sort into buckets

Every line below is part of the definition of convex function or of positive semidefinite (PSD) — one or the other, never both. Put each where it belongs.

convex function

A function whose value at any weighted average of two points is at most the same weighted average of the two function values.; the graph never rises above a chord.; α=1 is the x end, α=0 is the y end, α=½ is the midpoint.

positive semidefinite (PSD)

A symmetric matrix H with zᵀHz ≥ 0 for every vector z; equivalently, all eigenvalues ≥ 0.; It means non-negative curvature in every direction

b1

A function whose value at any weighted average of two points is at most the same weighted average of the two function values. Geometrically: the graph never rises above a chord. α sweeps the chord: α=1 is the x end, α=0 is the y end, α=½ is the midpoint.

b2

A symmetric matrix H with zᵀHz ≥ 0 for every vector z — equivalently, all eigenvalues ≥ 0. It means non-negative curvature in every direction: the surface never bends downward. This is the easiest of the three tests to check, since it is just an eigenvalue sign check.

Put in order

Put the moves of Compute the Hessian of our bowl by hand into the order they have to happen.

1. First partials
2. Second partials
3. Verify: eigenvalues are 1 and 4, both ≥ 0

Why: These are the moves of the worked example in the order it makes them, and each one is set up by the one before it. ∂f/∂w₁ = ½·2w₁ = w₁ and ∂f/∂w₂ = ½·8w₂ = 4w₂.

Annotate

From Compute the Hessian of our bowl by hand — read this one piece at a time. What is each part doing?

On: \( \nabla f(w) = \begin{bmatrix} w_1 \\ 4 w_2 \end{bmatrix} \)

• ∂f/∂w₁ = ½·2w₁ = w₁ and ∂f/∂w₂ = ½·8w₂ = 4w₂. So ∇f = [w₁, 4w₂].
• ∂²f/∂w₁² = 1, ∂²f/∂w₂² = 4, and the mixed partial ∂²f/∂w₁∂w₂ = 0 since w₁ and w₂ never multiply. The Hessian is constant — it does not depend on w.
• A diagonal matrix's eigenvalues are its diagonal entries. Both positive ⇒ PSD (in fact positive definite) at every w ⇒ f is convex everywhere. Certificate obtained.

Predict first

The table runs: Hessian ∇²f | [[1, 0], [0, 4]] (constant in w) · eigenvalues | [1.0, 4.0]

In Confirm the Hessian test in code, given the rows so far: what is the next one — the row where quantity is all ≥ 0??

Correct: all ≥ 0? | True → PSD → convex everywhere

Why: The relationship between the columns, not the individual numbers, is what generates the next row. The Hessian is constant, so this one eigenvalue check certifies convexity over the ENTIRE domain — no need to re-check at other points.

Comparison matrix

From Confirm the Hessian test in code: refill the value (verified) column from what you know. The rest of the table is as it appeared.

Predict first

A student writes this, and it looks reasonable:

To test convexity, find the critical point and check the Hessian there. If ∇²f ⪰ 0 at that point, call f convex.

It is wrong. Say what breaks — and say it before you turn the page.

Correct: A PSD Hessian at ONE point only certifies a LOCAL minimum.

Convexity is a global property. The Hessian must be PSD at every point in the domain, not just at the minimum.

Why: A PSD Hessian at ONE point only certifies a LOCAL minimum. The function is free to curve downward somewhere else and still pass this local test — so you can wrongly certify a non-convex function.

Discussion prompt

Its second derivative changes sign — before any calculation: what is the plan? Name the moves in order, in plain English, without doing the arithmetic.

Hint: It starts with: First derivative

Draw it

Draw what Its second derivative changes sign just did — the shape of it, not the line-by-line working. One picture, labels only where you need them. Then check it against the steps: anything you could not draw is a step you followed rather than understood.

Sort into buckets

Sort these by curvature, from memory, without looking back at Trace the curvature and catch the chord violation. Telling them apart on the spot is the skill; the table is only where the answer happens to be written down.

positive

−1.5; −1.0; 1.0; 1.5

negative

−0.5; 0.0; 0.5

g1

curvature is "positive" for −1.5, −1.0, 1.0, 1.5 — that is what the table on "Trace the curvature and catch the chord…" records, and it is the single property separating this group from the rest.

g2

curvature is "negative" for −0.5, 0.0, 0.5 — that is what the table on "Trace the curvature and catch the chord…" records, and it is the single property separating this group from the rest.

Comparison matrix

From Two minima — the trap made concrete: every row here is a choice with a cost. Fill the type column, then say which row you would actually pick and what you give up for it.

Discussion prompt

Prove: ∇f(x)=0 forces a global minimum — before any calculation: what is the plan? Name the moves in order, in plain English, without doing the arithmetic.

Hint: It starts with: Start from the first-order condition

Discussion prompt

Work backwards. The example finished here:

Verify the conclusion holds for every y

What was it asked to do, and what must it have been given? Reconstruct the problem from its answer.

Hint: Every quantity in the result had to enter somewhere. Account for each one.

Sort into buckets

These are the pieces of Lesson 11: Convex Optimization, out of order. Put each one back under the part of the lesson it belongs to.

What makes a problem convex

Our running example: a quadratic bowl; Convex = one bowl, no dents; The chord definition

When convexity fails

A function that is not convex; Its second derivative changes sign; Trace the curvature and catch the chord violation

Why convex ⇒ only global minima

The one theorem that pays for everything; Prove: ∇f(x)=0 forces a global minimum; What the proof means for training

s1

What makes a problem convex is where Lesson 11: Convex Optimization puts Our running example: a quadratic bowl, Convex = one bowl, no dents, The chord definition. Knowing which part of the lesson a problem belongs to is most of knowing which method to reach for.

s2

When convexity fails is where Lesson 11: Convex Optimization puts A function that is not convex, Its second derivative changes sign, Trace the curvature and catch the chord violation. Knowing which part of the lesson a problem belongs to is most of knowing which method to reach for.

s3

Why convex ⇒ only global minima is where Lesson 11: Convex Optimization puts The one theorem that pays for everything, Prove: ∇f(x)=0 forces a global minimum, What the proof means for training. Knowing which part of the lesson a problem belongs to is most of knowing which method to reach for.

Predict first

A student writes this, and it looks reasonable:

Like any function, a convex one might have local minima that trap gradient descent, so we still have to worry about initialization on linear regression.

It is wrong. Say what breaks — and say it before you turn the page.

Correct: This treats a convex bowl like a neural-net landscape.

A convex function has no local minimum other than the global one. Convergence is about speed, not about getting stuck.

Why: This treats a convex bowl like a neural-net landscape. For a strictly convex loss there is EXACTLY one valley — restarts all converge to the same point. The extra runs waste compute and learn nothing.

Discussion prompt

The rule this trap just fixed:

We just proved it: ∇f(x)=0 ⇒ f(y) ≥ f(x) for all y. So the only thing left to tune is the learning rate — which controls how FAST you reach the one minimum, the subject of Parts 4–5.

Now break it on purpose. Build a case that violates it and follow the consequences until something visibly fails. Where does the failure first show up — and would you have noticed it if you had not been looking?

Hint: The dangerous rules are the ones whose violation still produces an answer. If yours fails loudly, try to find one that fails quietly.

Fill in the blanks

From The MSE loss is convex — tying back to Lesson 7 — one line has had its right-hand side removed. Put it back.

import numpy as np
x = np.array([1.,2.,3.,4.,5.]); y = np.array([2.,4.,5.,4.,5.])
X = np.c_[np.ones(5), x]
H = **2 * X.T @ X** # Hessian of ||Xw - y||^2
print(H) # [[ 10. 30.] [ 30. 110.]]
print(np.linalg.eigvalsh(H)) # [ 1.690481 118.309519]

Why: H is what everything below it consumes, so the wrong expression here fails later and somewhere else. zᵀ(2XᵀX)z = 2‖Xz‖² ≥ 0 for any z, so 2XᵀX is PSD by construction — and it is CONSTANT in w.

Put in order

Put the moves of Prove 2XᵀX is PSD without eigenvalues into the order they have to happen.

1. Write out zᵀ(2XᵀX)z
2. Recognize a squared norm
3. Verify against the numbers

Why: These are the moves of the worked example in the order it makes them, and each one is set up by the one before it. Pull the constant 2 out front and group the middle: zᵀXᵀXz = (Xz)ᵀ(Xz).

Draw it

Translate both ways. First write the expression above as a sentence with no symbols in it at all. Then cover it, and write your sentence back as notation. If the two versions disagree, the disagreement is the thing to fix.

Discussion prompt

The 1-D shadow: derive the update factor — before any calculation: what is the plan? Name the moves in order, in plain English, without doing the arithmetic.

Hint: It starts with: Plug f'(x)=2x into the update

Draw it

Draw what The 1-D shadow: derive the update factor just did — the shape of it, not the line-by-line working. One picture, labels only where you need them. Then check it against the steps: anything you could not draw is a step you followed rather than understood.

Predict first

Start at x₀=2 and run six steps for η ∈ {0.1, 0.5, 0.9, 1.1}. Watch the contraction factor decide each fate. Runnable:

Commit before you compute: what does Four learning rates on f(x)=x² come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: η=0.5 lands at 0 in ONE step; η=1.1 blows up

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. η=0.5 gives factor 0 — the perfect step for this f.

Fill in the blanks

From Full trace: loss and gradient at η = 0.1 — one line has had its right-hand side removed. Put it back.

x, lr = 2.0, 0.1
for t in range(6):
print(t, round(x,5), round(xx,6), round(2x,5)) # t, x, f=x^2, grad=2x
x = *x - lr(2x)* # step downhill

Why: x is what everything below it consumes, so the wrong expression here fails later and somewhere else. Near the minimum the gradient fades, so each step gets smaller on its own — GD naturally decelerates as it homes in.

Step through it

Step through Full trace: loss and gradient at η = 0.1 one row at a time. What is driving the change, and what would the row after the last one be?

1. Step 1: step t is 0
2. Step 2: step t is 1
3. Step 3: step t is 2
4. Step 4: step t is 3
5. Step 5: step t is 4
6. Step 6: step t is 5

Predict first

A student writes this, and it looks reasonable:

Bigger steps cover more ground, so cranking η up must reach the minimum sooner. Use η = 1.1 on f(x)=x².

It is wrong. Say what breaks — and say it before you turn the page.

Correct: Above the stability threshold each step OVERSHOOTS the minimum by more than it started: 2 → −2.4 → 2.88 → −3.456 → … The iterate grows without bound.

There is a ceiling. Stay inside 0 < η < 1 (here), and the fastest step is the one that makes the factor smallest — η = 0.5 (factor 0).

Why: Above the stability threshold each step OVERSHOOTS the minimum by more than it started: 2 → −2.4 → 2.88 → −3.456 → … The iterate grows without bound. Bigger is not faster — bigger is broken.

Sort into buckets

Some of these hold up and some are the exact mistakes this lesson is built to prevent. Sort them.

Holds up

We carry one function through the whole lesson so every slide compounds. It is a 2-variable quadratic in w = (w₁, w₂):; Picture the graph of a function as a landscape. Convex means it is a single smooth bowl: no separate valleys, no dents, no hidden pockets to fall into.; Flip the sag upward and you get the non-convex double well of Part 2, where a chord can dip under the curve. One picture, two fates.

Breaks

To test convexity, find the critical point and check the Hessian there. If ∇²f ⪰ 0 at that point, call f convex.; Like any function, a convex one might have local minima that trap gradient descent, so we still have to worry about initialization on linear regression.

sound

These are stated as this lesson states them — each one survives the edge cases Lesson 11: Convex Optimization puts it through.

flawed

Each of these is lifted from a trap in this deck: reasonable-sounding, and wrong in a way that only shows up once you rely on it.

Discussion prompt

Explain Smoothness L bounds the curvature to a student a year behind you. No notation, no jargon they have not met — and it still has to be true.

Hint: If your explanation needs a symbol they have never seen, you are describing the notation rather than the idea.

Put in order

Put the moves of Derive the stable-step bound η < 2/L into the order they have to happen.

1. Require the contraction factor < 1 in magnitude
2. Unfold the absolute value
3. Verify: the tightest direction is λ = L

Why: These are the moves of the worked example in the order it makes them, and each one is set up by the one before it. Convergence in a direction with curvature λ needs |1 − ηλ| < 1, exactly the 1-D condition applied per eigen-direction.

Annotate

From Derive the stable-step bound η < 2/L — read this one piece at a time. What is each part doing?

On: \( \boxed{\,0 < \eta < \frac{2}{L}, \qquad L = \lambda_{\max}(\nabla^2 f)\,} \)

• Convergence in a direction with curvature λ needs |1 − ηλ| < 1, exactly the 1-D condition applied per eigen-direction.
• |1 − ηλ| < 1 means −1 < 1 − ηλ < 1. The right inequality gives ηλ > 0 (always true for η,λ > 0); the left gives ηλ < 2.
• The bound must hold for EVERY eigen-direction at once, so the smallest ceiling wins — the one from the largest λ, namely L. Hence the single stable-step rule below.

Predict first

The table runs: 0.2 | [1.049, 0.000] | +0.20 | converges · 0.5 | [0.062, 1.000] | −1.00 | w₂ stalls (|factor|=1)

In The bound on our 2-D bowl, given the rows so far: what is the next one — the row where η is 0.6?

Correct: 0.6 | [0.016, 7.530] | −1.40 | DIVERGES in w₂

Why: The relationship between the columns, not the individual numbers, is what generates the next row. The w₂ curvature is L=4, so its factor is 1−0.6·4 = −1.4, magnitude > 1: w₂ explodes (1 → −1.4 → 1.96 → …) even though w₁'s factor 1−0.6·1 = 0.4 is fine.

Discussion prompt

Explain Conditioning κ = L/μ sets the pace by analogy to something with no Machine Learning in it at all — a queue, a recipe, a map, a bank balance, whatever fits. Then say where your analogy breaks.

Hint: An analogy that never breaks is not an analogy, it is the same idea wearing a hat. Find the seam — that is the part that is actually new.

Discussion prompt

When μ > 0 (a curvature floor — 'strongly convex'), the loss gap shrinks by a constant ratio every step. On our bowl at η = 1/L = 0.25, the per-step factor settles to (1 − μ/L)² = (1 − 0.25)² = 0.5625. Runnable:

What do you need to know — or decide — before the first line can be written? List everything the problem has to hand you.

Hint: Anything you would have to invent to get started is a thing the problem must supply.

Step through it

Step through Strong convexity gives geometric decay one row at a time. What is driving the change, and what would the row after the last one be?

1. Step 1: step t is 1
2. Step 2: step t is 2
3. Step 3: step t is 3
4. Step 4: step t is 4
5. Step 5: step t is 5

Comparison matrix

From The convergence-rate classes: refill the meaning column from what you know. The rest of the table is as it appeared.

Predict first

Compare plain GD against heavy-ball momentum on f(w)=½(w₁²+4w₂²), both from w=(4,1), counting steps to drive the loss below 1e-8. Runnable:

Commit before you compute: what does Momentum beats plain GD on our bowl come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: Momentum reaches 1e-8 in 12 steps vs plain GD's 21

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. Heavy-ball with η=0.4444, β=0.1111 nearly halves the step count on this mild κ=4 bowl.

Discussion prompt

Work backwards. The example finished here:

Momentum reaches 1e-8 in 12 steps vs plain GD's 21

What was it asked to do, and what must it have been given? Reconstruct the problem from its answer.

Hint: Every quantity in the result had to enter somewhere. Account for each one.

Discussion prompt

Run Diagnosing any optimization problem with this step confiscated:

Conditioning κ = L/μ: large κ ⇒ slow zig-zagging → rescale features or precondition to shrink it.

Is it still possible? If it is, say what takes its place and what it costs you. If it is not, say exactly what that step was providing that nothing else does.

Hint: A step you can drop for free was never load-bearing. If you cannot drop it, name the thing that goes wrong the moment it is gone.

Discussion prompt

Diagnosing any optimization problem works on the cases you have just seen. Push it to the edge: what is the most degenerate input it still handles — empty, zero, one item, everything equal — and what is the first case where it stops being true? Name the case, not just "it breaks".

Hint: Try the smallest legal input, then the largest, then the one where two things collide. Methods are specified at their edges; the middle takes care of itself.

Eliminate the wrong options

A twice-differentiable f is convex if and only if its Hessian is…

3 of these 4 are wrong. Strike them one at a time, and say what rules each one out before you strike the next. The survivor is the answer.

• A. positive semidefinite at every point in the domain
• B. positive semidefinite at the minimum
• C. invertible everywhere
• D. symmetric

Survives elimination: A

Why: Convexity is equivalent to ∇²f ⪰ 0 everywhere — non-negative curvature in every direction at every point. Then every stationary point is a global minimum. For a constant Hessian (like diag(1,4)) one eigenvalue check settles it for all w.

Check your understanding

A twice-differentiable f is convex if and only if its Hessian is…

• A. positive semidefinite at every point in the domain (correct)
• B. positive semidefinite at the minimum
• C. invertible everywhere
• D. symmetric

Answer: A

Why: Convexity is equivalent to ∇²f ⪰ 0 everywhere — non-negative curvature in every direction at every point. Then every stationary point is a global minimum. For a constant Hessian (like diag(1,4)) one eigenvalue check settles it for all w.

Why B tempts people

PSD only at the minimum certifies a LOCAL minimum. g(t)=t⁴−3t² is convex near each of its two minima yet non-convex overall — exactly the trap we dissected.

Why C tempts people

Invertibility (no zero eigenvalue) is about strict/positive-definite convexity, and isn't required — a convex function can have a singular PSD Hessian with a flat direction.

Why D tempts people

Hessians are ALWAYS symmetric (mixed partials commute), so symmetry says nothing about convexity — it can't distinguish a bowl from a saddle.

Predict first

Gradient descent is run on a strictly convex loss from two different starting points (both with a stable learning rate). What happens?

Answer it in your own words, now, with nothing to choose from. The options are on the next slide — and picking the right one off a list is an easier skill than producing it.

Correct: Both converge to the same global minimum

Why: A strictly convex function has a single global minimum and no other stationary points, so any convergent run reaches the same point regardless of initialization — the ∇f(x)=0 ⇒ global proof guarantees it.

Check your understanding

Gradient descent is run on a strictly convex loss from two different starting points (both with a stable learning rate). What happens?

• A. Both converge to the same global minimum (correct)
• B. Each may land in a different local minimum
• C. One will diverge, the other converge
• D. They converge only if started close together

Answer: A

Why: A strictly convex function has a single global minimum and no other stationary points, so any convergent run reaches the same point regardless of initialization — the ∇f(x)=0 ⇒ global proof guarantees it.

Why B tempts people

Multiple local minima are a NON-convex phenomenon (like g(t)=t⁴−3t² or a neural net). A strictly convex function has exactly one — restarts are pointless.

Why C tempts people

Divergence is set by the learning rate (η vs 2/L), not the start point. With a stable η both converge; with η too large both diverge — the start doesn't decide it.

Why D tempts people

Convexity makes the result independent of where you start; proximity to the optimum is never required for convergence, only a stable step size.

Predict first

For f(x) = x² (so f''=2, hence L=2), which learning rate makes gradient descent DIVERGE?

Answer it in your own words, now, with nothing to choose from. The options are on the next slide — and picking the right one off a list is an easier skill than producing it.

Correct: η = 1.1

Why: The update multiplies x by (1−2η). Stability needs |1−2η| < 1, i.e. 0 < η < 1 = 2/L. η=1.1 gives factor −1.2, magnitude > 1, so |x| grows each step: 2 → −2.4 → 2.88 → … — divergence.

Check your understanding

For f(x) = x² (so f''=2, hence L=2), which learning rate makes gradient descent DIVERGE?

• A. η = 1.1 (correct)
• B. η = 0.5
• C. η = 0.1
• D. η = 0.9

Answer: A

Why: The update multiplies x by (1−2η). Stability needs |1−2η| < 1, i.e. 0 < η < 1 = 2/L. η=1.1 gives factor −1.2, magnitude > 1, so |x| grows each step: 2 → −2.4 → 2.88 → … — divergence.

Why B tempts people

η=0.5 gives factor 0 — it reaches the minimum in a single step. The fastest possible, the opposite of divergence.

Why C tempts people

η=0.1 gives factor +0.8 — slow but steadily converging, well inside the stable range 0 < η < 1.

Why D tempts people

η=0.9 gives factor |1−1.8| = 0.8 < 1 — it converges, just flipping sign each step. Under the η<1 threshold, so still stable.

Eliminate the wrong options

A convex quadratic has Hessian eigenvalues μ=1 and L=100. Its learning rate must satisfy η<2/L=0.02, capped by the STEEP direction. What limits convergence speed?

3 of these 4 are wrong. Strike them one at a time, and say what rules each one out before you strike the next. The survivor is the answer.

• A. The tiny μ direction: progress there scales with ημ ≤ 0.02, so κ=L/μ=100 makes it slow
• B. Nothing — a convex problem always converges in one step
• C. The steep L direction, which converges slowest
• D. The learning rate is irrelevant to speed once the problem is convex

Survives elimination: A

Why: η is capped at ~1/L by the steep direction, but that same small η makes progress in the shallow μ direction glacial: its factor is 1−ημ ≈ 1−μ/L = 0.99. The condition number κ = L/μ = 100 sets the pace — O(κ log 1/ε) steps.

Check your understanding

A convex quadratic has Hessian eigenvalues μ=1 and L=100. Its learning rate must satisfy η<2/L=0.02, capped by the STEEP direction. What limits convergence speed?

• A. The tiny μ direction: progress there scales with ημ ≤ 0.02, so κ=L/μ=100 makes it slow (correct)
• B. Nothing — a convex problem always converges in one step
• C. The steep L direction, which converges slowest
• D. The learning rate is irrelevant to speed once the problem is convex

Answer: A

Why: η is capped at ~1/L by the steep direction, but that same small η makes progress in the shallow μ direction glacial: its factor is 1−ημ ≈ 1−μ/L = 0.99. The condition number κ = L/μ = 100 sets the pace — O(κ log 1/ε) steps.

Why B tempts people

One-step convergence happens only when κ=1 (a round bowl). Here κ=100, so it takes on the order of 100·log(1/ε) steps.

Why C tempts people

The steep L direction actually converges FASTEST (its factor is smallest under η≈1/L). It's the shallow μ direction that lags — the common mix-up.

Why D tempts people

η is central to speed: too small crawls, too large (>2/L) diverges. Convexity guarantees you CAN converge, not that η stops mattering.

Discussion prompt

Build rules: type every line yourself, print the whole trajectory (don't just trust the final number), and watch the sign flip when η pushes past the stable range.

That is stated as though it always holds. Do one of two things: produce a case where it fails, or say precisely what rules such a case out. "It just does" is not on the menu.

Hint: Hunt at the extremes first — zero, one, negative, empty, equal. If every extreme survives, the reason they survive is the proof.

Step through it

Step through Milestone 2 — the GD loop one row at a time. What is driving the change, and what would the row after the last one be?

1. Step 1: step is 0
2. Step 2: step is 1
3. Step 3: step is 2
4. Step 4: step is 3
5. Step 5: step is 6

Comparison matrix

From Milestone 4 — the full program: every row here is a choice with a cost. Fill the verdict column, then say which row you would actually pick and what you give up for it.

Comparison matrix

From Stretch: watch the 2-D bound bite: refill the w = [w₁, w₂] column from what you know. The rest of the table is as it appeared.

Draw it

One page, no notation unless you need it: draw how these connect — What makes a problem convex · When convexity fails · Why convex ⇒ only global minima · Gradient descent as a contraction · The step-size bound η < 2/L · Your turn: GD convergence. Put an arrow wherever one of them is what makes another possible, and label the arrow with why.