Lesson 24: Constrained Optimization & KKT

Discussion prompt

Before we open Lesson 24: Constrained Optimization & KKT: without looking back, what was the main idea of The Multivariate Gaussian, and what could you do by the end of it that you could not do before?

Hint: One sentence for the idea, one for the skill. If the second one is blank, that is the part to revisit.

Discussion prompt

Unconstrained optimization asks 'where is f smallest?'. Constrained optimization adds rules the answer must respect — equalities h(x)=0 and inequalities g(x)≤0.

That is stated as though it always holds. Do one of two things: produce a case where it fails, or say precisely what rules such a case out. "It just does" is not on the menu.

Hint: Hunt at the extremes first — zero, one, negative, empty, equal. If every extreme survives, the reason they survive is the proof.

Discussion prompt

Explain Our running example by analogy to something with no Machine Learning in it at all — a queue, a recipe, a map, a bank balance, whatever fits. Then say where your analogy breaks.

Hint: An analogy that never breaks is not an analogy, it is the same idea wearing a hat. Find the seam — that is the part that is actually new.

Discussion prompt

Explain The unconstrained answer is usually illegal to a student a year behind you. No notation, no jargon they have not met — and it still has to be true.

Hint: If your explanation needs a symbol they have never seen, you are describing the notation rather than the idea.

Discussion prompt

Read the picture before the words. What is this showing, and what is the one thing it is built to make obvious? Commit to an answer, then read on.

Hint: Name the parts, then say what changes between them — and if nothing changes, say what is being held still.

Comparison matrix

From Two ways a constraint can bite: refill the optimum sits column from what you know. The rest of the table is as it appeared.

Put in order

Put the moves of Stationarity in x recovers tangency into the order they have to happen.

1. Differentiate L with respect to the x-variables
2. Set it to zero
3. This is exactly the tangency condition

Why: These are the moves of the worked example in the order it makes them, and each one is set up by the one before it. ∂/∂x of f(x)+λ h(x) is ∇f + λ∇h.

Annotate

From Stationarity in x recovers tangency — read this one piece at a time. What is each part doing?

On: \( \nabla_x \mathcal{L} = \nabla f(x) + \lambda\, \nabla h(x) \)

• ∂/∂x of f(x)+λ h(x) is ∇f + λ∇h. λ is treated as a constant here.
• At a stationary point the gradient vanishes.
• The gradients-parallel picture from Part 1 falls out with no extra work — the Lagrangian's x-stationarity IS 'aligned gradients'.

Discussion prompt

Stationarity in λ recovers the constraint — before any calculation: what is the plan? Name the moves in order, in plain English, without doing the arithmetic.

Hint: It starts with: Differentiate L with respect to λ

Draw it

Translate both ways. First write the expression above as a sentence with no symbols in it at all. Then cover it, and write your sentence back as notation. If the two versions disagree, the disagreement is the thing to fix.

Put in order

Put the moves of Solve our running example by hand into the order they have to happen.

1. ∂L/∂x = 2x + λ = 0 and ∂L/∂y = 2y + λ = 0
2. ∂L/∂λ = x + y − 1 = 0
3. Back out λ and f⋆

Why: These are the moves of the worked example in the order it makes them, and each one is set up by the one before it. Both give the same relation to λ, so 2x = 2y ⇒ x = y.

Discussion prompt

Work backwards. The example finished here:

Back out λ and f⋆

What was it asked to do, and what must it have been given? Reconstruct the problem from its answer.

Hint: Every quantity in the result had to enter somewhere. Account for each one.

Predict first

Hand it the same problem numerically. scipy.optimize.minimize takes each constraint as a dict; 'eq' means fun(v)=0. Runnable as-is:

Commit before you compute: what does Verify the hand answer against scipy come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: scipy lands on the same point

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. Prints [0.5 0.5] 0.5 — identical to the by-hand (0.5, 0.5) with f⋆ = 0.5.

Comparison matrix

From Verify the hand answer against scipy: every row here is a choice with a cost. Fill the by hand column, then say which row you would actually pick and what you give up for it.

Predict first

A student writes this, and it looks reasonable:

It's still x²+y², so minimize f directly and report the origin.

It is wrong. Say what breaks — and say it before you turn the page.

Correct: But (0,0) gives x+y = 0 ≠ 1 — it is OFF the line, i.e.

Fold the constraint in with a multiplier and solve ∇L = 0 and h = 0 together.

Why: But (0,0) gives x+y = 0 ≠ 1 — it is OFF the line, i.e. infeasible. The unconstrained minimum is almost always outside the feasible set, which is the entire reason constrained optimization exists.

Discussion prompt

Solve the problem at c = 1.00 and c = 1.01, then estimate df⋆/dc by finite difference. It should come out near 1 = −λ:

What do you need to know — or decide — before the first line can be written? List everything the problem has to hand you.

Hint: Anything you would have to invent to get started is a thing the problem must supply.

Sort into buckets

Every line below is part of the definition of feasible set or of active vs inactive — one or the other, never both. Put each where it belongs.

feasible set

The set of all x that satisfy every constraint.; Optimization now happens only inside this set; the true minimum of f may lie outside it, and then it is simply not allowed.

active vs inactive

A constraint is ACTIVE at x⋆ if g(x⋆)=0 (the solution sits on the boundary, the constraint is pushing back).; It is INACTIVE if g(x⋆)<0 (satisfied with room to spare, exerting no force).

b1

The set of all x that satisfy every constraint. Optimization now happens only inside this set — the true minimum of f may lie outside it, and then it is simply not allowed.

b2

A constraint is ACTIVE at x⋆ if g(x⋆)=0 (the solution sits on the boundary, the constraint is pushing back). It is INACTIVE if g(x⋆)<0 (satisfied with room to spare, exerting no force).

Sort into buckets

These are the pieces of Lesson 24: Constrained Optimization & KKT, out of order. Put each one back under the part of the lesson it belongs to.

The problem: rules to obey

The constrained problem; Our running example; The unconstrained answer is usually illegal

The Lagrangian

Package it into one function; Stationarity in x recovers tangency; Stationarity in λ recovers the constraint

Inequalities & KKT

Inequalities change the game; The four KKT conditions; Why the multiplier is one-signed now

s1

The problem: rules to obey is where Lesson 24: Constrained Optimization & KKT puts The constrained problem, Our running example, The unconstrained answer is usually illegal. Knowing which part of the lesson a problem belongs to is most of knowing which method to reach for.

s2

The Lagrangian is where Lesson 24: Constrained Optimization & KKT puts Package it into one function, Stationarity in x recovers tangency, Stationarity in λ recovers the constraint. Knowing which part of the lesson a problem belongs to is most of knowing which method to reach for.

s3

Inequalities & KKT is where Lesson 24: Constrained Optimization & KKT puts Inequalities change the game, The four KKT conditions, Why the multiplier is one-signed now. Knowing which part of the lesson a problem belongs to is most of knowing which method to reach for.

Predict first

The table runs: x⋆, y⋆ | [1, 0] | on the boundary · g = 1 − x at x⋆ | 0 | ACTIVE

In An ACTIVE inequality, solved and checked, given the rows so far: what is the next one — the row where quantity is μ?

Correct: μ | 2 > 0 | dual-feasible, may push

Why: The relationship between the columns, not the individual numbers, is what generates the next row. Prints [1. -0.] 1.0 and active: True.

Fill in the blanks

From All four KKT conditions on numbers — one line has had its right-hand side removed. Put it back.

import numpy as np
xs, ys, mu = 1.0, 0.0, 2.0 # solution + multiplier
grad_f = np.array([2xs, 2ys]) # grad of x^2+y^2
grad_g = np.array([-1.0, 0.0]) # g(x)=1-x, so grad_g=(-1,0)
print('stationarity :', grad_f + mu*grad_g) # want [0,0]
print('primal feas :', 1 - xs, '<= 0') # g <= 0
print('dual feas :', mu, '>= 0') # mu >= 0
print('slackness :', mu * (1 - xs), '= 0') # mu*g = 0

Why: grad_g is what everything below it consumes, so the wrong expression here fails later and somewhere else. Stationarity (2,0)+2(−1,0) = [0,0]; primal 1−1 = 0 ≤ 0; dual 2 ≥ 0; slackness 2·0 = 0.

Predict first

A student writes this, and it looks reasonable:

There are constraints in the problem, so give each one a positive multiplier μᵢ > 0.

It is wrong. Say what breaks — and say it before you turn the page.

Correct: An INACTIVE constraint (gᵢ < 0, slack) has μᵢ = 0 — complementary slackness μᵢ gᵢ = 0 with gᵢ ≠ 0 FORCES μᵢ = 0.

Let complementary slackness sort constraints into active (μ>0) and inactive (μ=0).

Why: An INACTIVE constraint (gᵢ < 0, slack) has μᵢ = 0 — complementary slackness μᵢ gᵢ = 0 with gᵢ ≠ 0 FORCES μᵢ = 0. Assuming all μᵢ > 0 gives an over-determined, inconsistent system.

Discussion prompt

The rule this trap just fixed:

Let complementary slackness sort constraints into active (μ>0) and inactive (μ=0).

Now break it on purpose. Build a case that violates it and follow the consequences until something visibly fails. Where does the failure first show up — and would you have noticed it if you had not been looking?

Hint: The dangerous rules are the ones whose violation still produces an answer. If yours fails loudly, try to find one that fails quietly.

Step through it

Step through Our running SVM: four points on a line one row at a time. What is driving the change, and what would the row after the last one be?

1. Step 1: point x is 1
2. Step 2: point x is 2
3. Step 3: point x is 4
4. Step 4: point x is 5

Put in order

Put the moves of Solve the SVM primal by hand into the order they have to happen.

1. Subtract the two margin equations
2. Back-substitute for b
3. Half-margin = 1/‖w‖ = 1

Why: These are the moves of the worked example in the order it makes them, and each one is set up by the one before it. (w·4+b) − (w·2+b) = 1 − (−1) gives 2w = 2, so w = 1.

Annotate

From Solve the SVM primal by hand — read this one piece at a time. What is each part doing?

On: \( \boxed{\,w = 1, \quad b = -3, \quad \text{boundary at } x = 3\,} \)

• (w·4+b) − (w·2+b) = 1 − (−1) gives 2w = 2, so w = 1.
• From w·4 + b = 1 with w = 1: 4 + b = 1, so b = −3. Boundary w·x+b = 0 sits at x = 3, the midpoint — as expected.
• The margin planes sit at x = 2 and x = 4, distance 1 either side of x = 3. Full margin 2/‖w‖ = 2.

Predict first

A linear SVC with huge C approximates the hard margin. It should recover w=1, b=−3 and flag x=2, 4 as the support vectors:

Commit before you compute: what does Confirm the primal with scikit-learn come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: sklearn matches the hand answer exactly

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. Prints w = [1.], b = [-3.], support vectors x = [2.

Fill in the blanks

From Solve the SVM dual as a QP — one line has had its right-hand side removed. Put it back.

import numpy as np
from scipy.optimize import minimize
x = np.array([1., 2., 4., 5.])
y = np.array([-1., -1., 1., 1.])
G = np.outer(y, y) * np.outer(x, x) # G_ij = y_i y_j x_i x_j
obj = lambda a: 0.5 * a @ G @ a - a.sum() # minimize -dual
cons = (**(a * y) @ x**,) # sum a_i y_i = 0
a = minimize(obj, np.zeros(4), bounds=[(0, None)]*4,
constraints=cons, method='SLSQP').x
print('alpha =', a.round(4))
w = ___
k = a.argmax()
b = y[k] - w * x[k]
print('w =', round(w, 4), ' b =', round(b, 4))

Why: w is what everything below it consumes, so the wrong expression here fails later and somewhere else. Prints alpha = [0. 0.5 0.5 0.] and w = 1.0, b = -3.0 — identical to the primal and to sklearn.

Comparison matrix

From Solve the SVM dual as a QP: refill the label y column from what you know. The rest of the table is as it appeared.

Predict first

The table runs: 1 | 0.0 | 2.0 | 0.0 (inactive) · 2 | 0.5 | 1.0 | 0.0 (active SV) · 4 | 0.5 | 1.0 | 0.0 (active SV)

In Complementary slackness makes it sparse, given the rows so far: what is the next one — the row where x is 5?

Correct: 5 | 0.0 | 2.0 | 0.0 (inactive)

Why: The relationship between the columns, not the individual numbers, is what generates the next row. Points x=2, 4 sit exactly ON the margin (margin = 1, constraint ACTIVE) and carry α = 0.5.

Discussion prompt

The dual came from two eliminations: w = Σ αᵢ yᵢ xᵢ and Σ αᵢ yᵢ = 0. Verify both hold at the solved α = [0, 0.5, 0.5, 0]:

What do you need to know — or decide — before the first line can be written? List everything the problem has to hand you.

Hint: Anything you would have to invent to get started is a thing the problem must supply.

Discussion prompt

Work backwards. The example finished here:

Both identities check out

What was it asked to do, and what must it have been given? Reconstruct the problem from its answer.

Hint: Every quantity in the result had to enter somewhere. Account for each one.

Predict first

A student writes this, and it looks reasonable:

The extreme points x=1 and x=5 are the most 'characteristic' of each class, so they must be the support vectors.

It is wrong. Say what breaks — and say it before you turn the page.

Correct: those are the points FARTHEST from the boundary, with margin 2 > 1 — their constraints are inactive, so α = 0.

Support vectors are the points whose margin constraint is active (margin = 1).

Why: those are the points FARTHEST from the boundary, with margin 2 > 1 — their constraints are inactive, so α = 0. They could be deleted and the boundary would not move.

Sort into buckets

Some of these hold up and some are the exact mistakes this lesson is built to prevent. Sort them.

Holds up

Unconstrained optimization asks 'where is f smallest?'. Constrained optimization adds rules the answer must respect — equalities h(x)=0 and inequalities g(x)≤0.; One tiny problem will carry the entire Lagrange story. Minimize the squared distance to the origin, but you must stay on a line:; Picture concentric circles x²+y²=r² growing out from the origin — level sets of f. Small circles are cheap, big circles are expensive.

Breaks

It's still x²+y², so minimize f directly and report the origin.; There are constraints in the problem, so give each one a positive multiplier μᵢ > 0.

sound

These are stated as this lesson states them — each one survives the edge cases Lesson 24: Constrained Optimization & KKT puts it through.

flawed

Each of these is lifted from a trap in this deck: reasonable-sounding, and wrong in a way that only shows up once you rely on it.

Discussion prompt

Explain The dual function and weak duality to a student a year behind you. No notation, no jargon they have not met — and it still has to be true.

Hint: If your explanation needs a symbol they have never seen, you are describing the notation rather than the idea.

Predict first

Evaluate g(λ) = −λ²/2 − λ and maximize it. Its peak should equal the primal optimum p⋆ = 0.5 — a zero duality gap:

Commit before you compute: what does Strong duality: the gap is zero come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: Dual optimum = primal optimum

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. g peaks at λ = −1 with g(−1) = 0.5, exactly the primal f⋆ = 0.5.

Sort into buckets

Sort these by g(λ) (verified), from memory, without looking back at Strong duality: the gap is zero. Telling them apart on the spot is the skill; the table is only where the answer happens to be written down.

0.000

−2.0; 0.0

0.500

−1.0

0.375

−0.5

g1

g(λ) (verified) is "0.000" for −2.0, 0.0 — that is what the table on "Strong duality: the gap is zero" records, and it is the single property separating this group from the rest.

g2

g(λ) (verified) is "0.500" for −1.0 — that is what the table on "Strong duality: the gap is zero" records, and it is the single property separating this group from the rest.

g3

g(λ) (verified) is "0.375" for −0.5 — that is what the table on "Strong duality: the gap is zero" records, and it is the single property separating this group from the rest.

Put in order

These are the steps of The constrained-optimization recipe, scrambled. Put them back in order before the next slide shows you.

1. Form the Lagrangian L = f + Σⱼ λⱼ hⱼ + Σᵢ μᵢ gᵢ (one multiplier per constraint)
2. Stationarity: solve ∇ₓ L = 0 for the primal variables
3. Feasibility: enforce primal h=0, g≤0 and dual μᵢ ≥ 0
4. Complementary slackness: μᵢ gᵢ = 0 — split constraints into active (μ>0) and inactive (μ=0)
5. If convex + Slater, solve the dual instead — same optimum, and it exposes structure (support vectors, kernels)
6. Sanity-check with scipy/sklearn and read λ as the shadow price

Why: This is the order the recipe itself gives. Recalling the sequence without the slide in front of you is the difference between recognising the method and being able to run it — most of what goes wrong in practice is a step done out of turn.

Discussion prompt

The constrained-optimization recipe works on the cases you have just seen. Push it to the edge: what is the most degenerate input it still handles — empty, zero, one item, everything equal — and what is the first case where it stops being true? Name the case, not just "it breaks".

Hint: Try the smallest legal input, then the largest, then the one where two things collide. Methods are specified at their edges; the middle takes care of itself.

Eliminate the wrong options

At the solution of min f s.t. h(x)=0, stationarity gives ∇f + λ∇h = 0. What does this say geometrically?

3 of these 4 are wrong. Strike them one at a time, and say what rules each one out before you strike the next. The survivor is the answer.

• A. ∇f is parallel to ∇h, so no feasible (along-the-constraint) step lowers f
• B. ∇f = 0 at the solution
• C. The constraint can be dropped once you know λ
• D. λ must equal 0

Survives elimination: A

Why: ∇f = −λ∇h means ∇f points straight across the constraint surface (along ∇h). Every legal step runs along the surface, perpendicular to ∇f, so no feasible move decreases f — that is optimality on the constraint.

Check your understanding

At the solution of min f s.t. h(x)=0, stationarity gives ∇f + λ∇h = 0. What does this say geometrically?

• A. ∇f is parallel to ∇h, so no feasible (along-the-constraint) step lowers f (correct)
• B. ∇f = 0 at the solution
• C. The constraint can be dropped once you know λ
• D. λ must equal 0

Answer: A

Why: ∇f = −λ∇h means ∇f points straight across the constraint surface (along ∇h). Every legal step runs along the surface, perpendicular to ∇f, so no feasible move decreases f — that is optimality on the constraint.

Why B tempts people

∇f need NOT vanish at a constrained optimum — it is balanced by λ∇h, not zero. ∇f = 0 only if the constraint happens to be inactive (then λ = 0).

Why C tempts people

The λ∇h term IS the constraint's influence; dropping the constraint returns you to the (infeasible) unconstrained minimum at the origin.

Why D tempts people

λ is generally nonzero for a binding constraint — here λ = −1. λ = 0 would mean the constraint exerts no force.

Predict first

For min x²+y² s.t. x+y=c, the multiplier is λ=−1 at c=1. If you loosen the constraint to c=1.01, the optimal value f⋆ changes by approximately:

Answer it in your own words, now, with nothing to choose from. The options are on the next slide — and picking the right one off a list is an easier skill than producing it.

Correct: +0.01, because df⋆/dc = −λ = 1

Why: The envelope theorem gives df⋆/dc = −λ = 1, so a step of Δc = 0.01 raises f⋆ by about 0.01. Numerically f⋆ went 0.50000 → 0.51005, a rise of ≈ 0.01.

Check your understanding

For min x²+y² s.t. x+y=c, the multiplier is λ=−1 at c=1. If you loosen the constraint to c=1.01, the optimal value f⋆ changes by approximately:

• A. +0.01, because df⋆/dc = −λ = 1 (correct)
• B. −0.01, because loosening a constraint always lowers the cost
• C. 0, because λ does not affect the optimal value
• D. +1, because the shadow price is added directly to f⋆

Answer: A

Why: The envelope theorem gives df⋆/dc = −λ = 1, so a step of Δc = 0.01 raises f⋆ by about 0.01. Numerically f⋆ went 0.50000 → 0.51005, a rise of ≈ 0.01.

Why B tempts people

Loosening does not always lower cost — here the feasible line moves farther from the origin, so the minimum squared distance RISES. Sign of the change is set by λ, not by a blanket rule.

Why C tempts people

λ is exactly the sensitivity of the optimal value to the constraint level — that is the whole point of the shadow-price interpretation.

Why D tempts people

The shadow price is a RATE (df⋆/dc), not an amount added to f⋆. Multiply it by the step Δc = 0.01 to get the ≈ 0.01 change.

Predict first

An inequality constraint gᵢ(x) ≤ 0 is strictly satisfied at the optimum, gᵢ(x⋆) < 0. Its KKT multiplier μᵢ is:

Commit to an answer, then rate it — certain, fairly sure, or guessing — and write the rating down before you turn the page.

Correct: 0 — the constraint is inactive and exerts no force

Why: Complementary slackness μᵢ gᵢ = 0 with gᵢ < 0 forces μᵢ = 0. A constraint satisfied with room to spare pushes on nothing — delete it and the optimum is unchanged.

The rating matters as much as the answer: confident-and-wrong is the combination that survives revision, because nothing about it feels like it needs revisiting.

Check your understanding

An inequality constraint gᵢ(x) ≤ 0 is strictly satisfied at the optimum, gᵢ(x⋆) < 0. Its KKT multiplier μᵢ is:

• A. 0 — the constraint is inactive and exerts no force (correct)
• B. strictly positive
• C. negative
• D. equal to gᵢ(x⋆)

Answer: A

Why: Complementary slackness μᵢ gᵢ = 0 with gᵢ < 0 forces μᵢ = 0. A constraint satisfied with room to spare pushes on nothing — delete it and the optimum is unchanged.

Why B tempts people

A positive multiplier requires an ACTIVE constraint (gᵢ = 0). With gᵢ < 0 the product μᵢ gᵢ could not be zero unless μᵢ = 0.

Why C tempts people

Dual feasibility requires μᵢ ≥ 0; negative inequality multipliers are never allowed — a constraint can only push inward.

Why D tempts people

μᵢ and gᵢ are different objects (a price vs a slack). Slackness sets μᵢ = 0, not μᵢ = gᵢ.

Predict first

For our SVM the dual optimum equals the primal optimum. Which statement is correct in general?

Answer it in your own words, now, with nothing to choose from. The options are on the next slide — and picking the right one off a list is an easier skill than producing it.

Correct: Weak duality (dual ≤ primal) always holds; the gap is zero when the problem is convex and Slater's condition holds

Why: Weak duality g(λ) ≤ p⋆ is automatic (min-max ≥ max-min). The gap closes — strong duality — for convex problems with a strictly feasible point (Slater). The SVM is convex and strictly feasible on separable data, so its dual is exact.

Check your understanding

For our SVM the dual optimum equals the primal optimum. Which statement is correct in general?

• A. Weak duality (dual ≤ primal) always holds; the gap is zero when the problem is convex and Slater's condition holds (correct)
• B. The dual always equals the primal, for every optimization problem
• C. The dual is always an upper bound on the primal
• D. Strong duality requires the problem to be non-convex

Answer: A

Why: Weak duality g(λ) ≤ p⋆ is automatic (min-max ≥ max-min). The gap closes — strong duality — for convex problems with a strictly feasible point (Slater). The SVM is convex and strictly feasible on separable data, so its dual is exact.

Why B tempts people

For non-convex problems a strictly positive duality gap is common; equality is not guaranteed. It holds here only because the SVM is convex and satisfies Slater.

Why C tempts people

The dual is a LOWER bound (dual ≤ primal), because swapping min and max can only decrease the value. It is never guaranteed to be an upper bound.

Why D tempts people

Backwards: strong duality is guaranteed for CONVEX problems (with Slater). Non-convexity is exactly what can OPEN a gap.

Eliminate the wrong options

In our 4-point SVM (neg at x=1,2; pos at x=4,5) the dual gives α = [0, 0.5, 0.5, 0]. Why do exactly x=2 and x=4 have α > 0?

3 of these 4 are wrong. Strike them one at a time, and say what rules each one out before you strike the next. The survivor is the answer.

• A. Their margin constraints are active (they sit exactly on the margin), so slackness allows α > 0
• B. They are the points farthest from the boundary
• C. They are chosen at random by the QP solver
• D. Every point in each class contributes equally, and rounding zeroed two of them

Survives elimination: A

Why: x=2 and x=4 have margin y(wx+b) = 1 exactly — their constraints are active — so complementary slackness α(margin−1)=0 permits α > 0. They are the support vectors. x=1 and x=5 have margin 2 (inactive), forcing α = 0.

Check your understanding

In our 4-point SVM (neg at x=1,2; pos at x=4,5) the dual gives α = [0, 0.5, 0.5, 0]. Why do exactly x=2 and x=4 have α > 0?

• A. Their margin constraints are active (they sit exactly on the margin), so slackness allows α > 0 (correct)
• B. They are the points farthest from the boundary
• C. They are chosen at random by the QP solver
• D. Every point in each class contributes equally, and rounding zeroed two of them

Answer: A

Why: x=2 and x=4 have margin y(wx+b) = 1 exactly — their constraints are active — so complementary slackness α(margin−1)=0 permits α > 0. They are the support vectors. x=1 and x=5 have margin 2 (inactive), forcing α = 0.

Why B tempts people

The opposite: x=1 and x=5 are the FARTHEST (margin 2), and they get α = 0. Support vectors are the CLOSEST opposing points, on the margin.

Why C tempts people

The nonzero set is geometrically determined by which constraints are active, not random — rerunning gives the same [0, 0.5, 0.5, 0].

Why D tempts people

The values are not equal or rounded: x=1 and x=5 have exact α = 0 because their constraints are strictly slack, which is genuine sparsity, not a rounding artifact.

Discussion prompt

Explain Project: from Lagrange to the SVM dual by analogy to something with no Machine Learning in it at all — a queue, a recipe, a map, a bank balance, whatever fits. Then say where your analogy breaks.

Hint: An analogy that never breaks is not an analogy, it is the same idea wearing a hat. Find the seam — that is the part that is actually new.

Step through it

Step through Milestone 2 — an active inequality one row at a time. What is driving the change, and what would the row after the last one be?

1. Step 1: quantity is x⋆, y⋆
2. Step 2: quantity is f⋆
3. Step 3: quantity is active (x=1)?

Comparison matrix

From Milestone 3 — the SVM dual and its support vectors: every row here is a choice with a cost. Fill the value column, then say which row you would actually pick and what you give up for it.

Comparison matrix

From The full program: refill the value (verified) column from what you know. The rest of the table is as it appeared.

Discussion prompt

Slides closed, out loud: explain (1) why ∇f ∥ ∇h at an equality optimum, (2) what complementary slackness says about active vs inactive constraints, and (3) why only support vectors have α > 0.

That is stated as though it always holds. Do one of two things: produce a case where it fails, or say precisely what rules such a case out. "It just does" is not on the menu.

Hint: Hunt at the extremes first — zero, one, negative, empty, equal. If every extreme survives, the reason they survive is the proof.

Draw it

One page, no notation unless you need it: draw how these connect — The problem: rules to obey · The Lagrangian · Inequalities & KKT · The SVM, primal and dual · Duality & Slater · Recipe, checks & your turn. Put an arrow wherever one of them is what makes another possible, and label the arrow with why.