Lesson 17: Loss Functions & Their Gradients

Discussion prompt

Before we open Lesson 17: Loss Functions & Their Gradients: without looking back, what was the main idea of Singular Value Decomposition, and what could you do by the end of it that you could not do before?

Hint: One sentence for the idea, one for the skill. If the second one is blank, that is the part to revisit.

Discussion prompt

A model outputs a prediction f; the truth is y. A loss L(y, f) collapses the mismatch into a single non-negative number - 0 means perfect, bigger means worse.

That is stated as though it always holds. Do one of two things: produce a case where it fails, or say precisely what rules such a case out. "It just does" is not on the menu.

Hint: Hunt at the extremes first — zero, one, negative, empty, equal. If every extreme survives, the reason they survive is the proof.

Comparison matrix

From Our running examples: refill the target y column from what you know. The rest of the table is as it appeared.

Discussion prompt

Explain MSE: average the squared misses by analogy to something with no Machine Learning in it at all — a queue, a recipe, a map, a bank balance, whatever fits. Then say where your analogy breaks.

Hint: An analogy that never breaks is not an analogy, it is the same idea wearing a hat. Find the seam — that is the part that is actually new.

Predict first

Take y = [3, 5, 7], f = [2, 6, 4]. Residuals are [+1, -1, +3]. Square, sum, divide by n = 3:

Commit before you compute: what does MSE on our data, entry by entry come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: Sum then divide: (1 + 1 + 9) / 3 = 11/3

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. The mean keeps the loss on the same scale regardless of dataset size n, so learning rates transfer across datasets.

Comparison matrix

From MSE on our data, entry by entry: every row here is a choice with a cost. Fill the rᵢ = yᵢ - fᵢ column, then say which row you would actually pick and what you give up for it.

Put in order

Put the moves of Differentiate MSE - one prediction first into the order they have to happen.

1. Let u = yᵢ - fᵢ, so the term is u²/n
2. Inside derivative: d(yᵢ - fᵢ)/dfᵢ = -1
3. Stack all i into a vector

Why: These are the moves of the worked example in the order it makes them, and each one is set up by the one before it. Chain rule: d(u²)/df = 2u · du/df.

Annotate

From Differentiate MSE - one prediction first — read this one piece at a time. What is each part doing?

On: \( \frac{\partial}{\partial f_i}\,\frac{(y_i - f_i)^2}{n} = \frac{1}{n}\cdot 2(y_i - f_i)\cdot\frac{\partial(y_i - f_i)}{\partial f_i} \)

• Chain rule: d(u²)/df = 2u · du/df. We differentiate the outer square, then the inside.
• yᵢ is a constant target; only -fᵢ depends on fᵢ. That minus sign is the whole reason the gradient has a leading minus.
• Every coordinate has the same form, so the full gradient is one clean vector expression.

Discussion prompt

Compute the value and the gradient -2(y - f)/n directly. Self-contained - re-imports NumPy and re-defines y, f:

What do you need to know — or decide — before the first line can be written? List everything the problem has to hand you.

Hint: Anything you would have to invent to get started is a thing the problem must supply.

Step through it

Step through MSE gradient, verified in code one row at a time. What is driving the change, and what would the row after the last one be?

1. Step 1: quantity is MSE
2. Step 2: quantity is grad[0]
3. Step 3: quantity is grad[1]
4. Step 4: quantity is grad[2]

Discussion prompt

Explain Why the sign points the right way to a student a year behind you. No notation, no jargon they have not met — and it still has to be true.

Hint: If your explanation needs a symbol they have never seen, you are describing the notation rather than the idea.

Predict first

The table runs: 0 | +1 | 1 | +1 · 1 | -1 | 1 | -1

In MAE on our data, given the rows so far: what is the next one — the row where i is 2?

Correct: 2 | +3 | 3 | +1

Why: The relationship between the columns, not the individual numbers, is what generates the next row. Compare to MSE's numerator of 11: the big miss (+3) contributes 3 here versus 9 there.

Discussion prompt

Work backwards. The example finished here:

|+1| + |-1| + |+3| = 1 + 1 + 3 = 5

What was it asked to do, and what must it have been given? Reconstruct the problem from its answer.

Hint: Every quantity in the result had to enter somewhere. Account for each one.

Discussion prompt

Differentiate MAE - the sign appears — before any calculation: what is the plan? Name the moves in order, in plain English, without doing the arithmetic.

Hint: It starts with: d|u|/du = sign(u) for u ≠ 0

Draw it

Draw what Differentiate MAE - the sign appears just did — the shape of it, not the line-by-line working. One picture, labels only where you need them. Then check it against the steps: anything you could not draw is a step you followed rather than understood.

Fill in the blanks

From MAE gradient, verified in code — one line has had its right-hand side removed. Put it back.

import numpy as np
y = np.array([3., 5., 7.])
f = np.array([2., 6., 4.])
n = len(y)
mae = np.mean(np.abs(y - f))
grad = -np.sign(y - f)/n
print(round(mae, 4))
print(grad.round(4))

Why: n is what everything below it consumes, so the wrong expression here fails later and somewhere else. Every gradient entry is ±1/3. The big miss at i=2 gets the SAME magnitude as the small misses - unlike MSE, where it was -2.0.

Predict first

A student writes this, and it looks reasonable:

MAE is a simple linear-looking function of the error, so its gradient is just ±1/n everywhere, including when the prediction is exactly right.

It is wrong. Say what breaks — and say it before you turn the page.

Correct: At yᵢ = fᵢ the absolute value has a CORNER: the slope jumps from -1/n (left) to +1/n (right).

MAE is non-differentiable at 0. There, use a subgradient - any value in the interval [-1/n, +1/n].

Why: At yᵢ = fᵢ the absolute value has a CORNER: the slope jumps from -1/n (left) to +1/n (right). No single number is 'the' derivative there - the two-sided limits disagree.

Discussion prompt

The rule this trap just fixed:

Optimizers pick a value in [-1/n, 1/n] (numpy's sign(0) = 0) and march on. This same subgradient idea handles the hinge kink and L1 (Lesson 12) - a corner is fine, you just choose a slope.

Now break it on purpose. Build a case that violates it and follow the consequences until something visibly fails. Where does the failure first show up — and would you have noticed it if you had not been looking?

Hint: The dangerous rules are the ones whose violation still produces an answer. If yours fails loudly, try to find one that fails quietly.

Step through it

Step through The stability trick: subtract the max one row at a time. What is driving the change, and what would the row after the last one be?

1. Step 1: k is 0
2. Step 2: k is 1
3. Step 3: k is 2

Fill in the blanks

Finish softmax: normalize, with the scaffolding fading: two lines are gone now — fill both.

import numpy as np
def softmax(z):
e = np.exp(z - z.max())
return e / e.sum()
z = np.array([1., 2., 3.])
p = softmax(z)
print(p.round(4))
onehot = np.array([0., 0., 1.])
print((p - onehot).round(4))

Why: Reproducing these unaided, rather than reading them, is what tells you the method has transferred. Class 2 (the largest logit) gets the largest probability, 0.665.

Fill in the blanks

From Prove the max-subtraction is exact — one line has had its right-hand side removed. Put it back.

import numpy as np
def softmax(z):
e = np.exp(z - z.max()); return e / e.sum()
def naive(z):
e = np.exp(z); return e / e.sum()
z = np.array([1., 2., 3.])
print(softmax(z).round(6))
print(naive(z).round(6))
print(np.allclose(softmax(z), naive(z)))

Why: z is what everything below it consumes, so the wrong expression here fails later and somewhere else. Subtracting max(z) multiplies numerator and denominator by the same e^{-c}, which cancels - so the probabilities are identical, just computed without overflow risk.

Predict first

True class is 2, and p₂ = 0.6652. Cross-entropy is just -log(0.6652):

Commit before you compute: what does CE value on our logits come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: Sanity: if p_true were 1.0, CE = -log 1 = 0

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. A perfect confident prediction has zero loss; as p_true → 0 the loss → +infinity.

Discussion prompt

Piece 1: dL/dp for cross-entropy — before any calculation: what is the plan? Name the moves in order, in plain English, without doing the arithmetic.

Hint: It starts with: L = -Σ y_k log p_k

Put in order

Put the moves of Piece 2: the softmax Jacobian into the order they have to happen.

1. Diagonal k = j: ∂pₖ/∂zₖ = pₖ(1 - pₖ)
2. Off-diagonal k ≠ j: ∂pₖ/∂z_j = -pₖ p_j
3. Both cases in one line

Why: These are the moves of the worked example in the order it makes them, and each one is set up by the one before it. The numerator e^{zₖ} depends on zₖ too, so you get pₖ minus pₖ² from the quotient rule.

Draw it

Translate both ways. First write the expression above as a sentence with no symbols in it at all. Then cover it, and write your sentence back as notation. If the two versions disagree, the disagreement is the thing to fix.

Discussion prompt

Multiply the pieces - the cancellation — before any calculation: what is the plan? Name the moves in order, in plain English, without doing the arithmetic.

Hint: It starts with: Chain: ∂L/∂z_j = Σ_k (-y_k/p_k) · p_k(δ_kj - p_j)

Fill in the blanks

Verify p - y numerically (Jacobian route), with the scaffolding fading: two lines are gone now — fill both.

import numpy as np
z = np.array([1., 2., 3.])
e = np.exp(z - z.max())
p = e / e.sum()
J = np.diag(p) - np.outer(p, p)
onehot = np.array([0., 0., 1.])
dLdp = -onehot / p
dLdz = J.T @ dLdp
print(dLdz.round(4))
print((p - onehot).round(4))

Why: Reproducing these unaided, rather than reading them, is what tells you the method has transferred. The full chain-rule product equals p - y exactly, confirming the algebra.

Step through it

Step through Verify p - y numerically (Jacobian route) one row at a time. One of these columns never changes — find it, and say why it cannot.

1. Step 1: k is 0
2. Step 2: k is 1
3. Step 3: k is 2

Comparison matrix

From Verify against torch autograd: refill the y (one-hot) column from what you know. The rest of the table is as it appeared.

Fill in the blanks

From The binary case: sigmoid + BCE gives p - y too — one line has had its right-hand side removed. Put it back.

import torch
z = torch.tensor([0.5], requires_grad=True)
loss = torch.nn.functional.binary_cross_entropy_with_logits(z, torch.tensor([1.]))
loss.backward()
p = torch.sigmoid(z).item()
print(round(p, 4), round(z.grad.item(), 4), round(p - 1, 4))

Why: p is what everything below it consumes, so the wrong expression here fails later and somewhere else. Same clean form. The 0.3775 gap is exactly how far the prediction (0.6225) sits below the target (1.0).

Discussion prompt

Prove sigmoid + BCE = p - y by hand — before any calculation: what is the plan? Name the moves in order, in plain English, without doing the arithmetic.

Hint: It starts with: L = -y log p - (1-y) log(1-p), p = σ(z)

Draw it

Draw what Prove sigmoid + BCE = p - y by hand just did — the shape of it, not the line-by-line working. One picture, labels only where you need them. Then check it against the steps: anything you could not draw is a step you followed rather than understood.

Predict first

The table runs: -2.0 | 0.119203 | 0.119203 · 0.0 | 0.500000 | 0.500000 · 0.5 | 0.622459 | 0.622459

In Sigmoid IS the two-class softmax, given the rows so far: what is the next one — the row where z is 3.0?

Correct: 3.0 | 0.952574 | 0.952574

Why: The relationship between the columns, not the individual numbers, is what generates the next row. The two are algebraically identical, not merely close.

Step through it

Step through Sigmoid IS the two-class softmax and watch the numbers move. Now imagine the input ten times bigger: which column is the one that stops this being practical?

1. Step 1: z is -2.0
2. Step 2: z is 0.0
3. Step 3: z is 0.5
4. Step 4: z is 3.0

Discussion prompt

Explain The multiplier dies when you're confidently wrong to a student a year behind you. No notation, no jargon they have not met — and it still has to be true.

Hint: If your explanation needs a symbol they have never seen, you are describing the notation rather than the idea.

Discussion prompt

True label y = 1, but the logit is strongly negative (z = -4, so p ≈ 0.018). Compare the gradient each loss delivers. Self-contained:

What do you need to know — or decide — before the first line can be written? List everything the problem has to hand you.

Hint: Anything you would have to invent to get started is a thing the problem must supply.

Discussion prompt

Work backwards. The example finished here:

At z = -4: CE grad -0.982 (strong) vs MSE grad -0.017 (≈ 0)

What was it asked to do, and what must it have been given? Reconstruct the problem from its answer.

Hint: Every quantity in the result had to enter somewhere. Account for each one.

Predict first

A student writes this, and it looks reasonable:

A loss is a loss - MSE minimizes squared error, so just apply sigmoid + MSE to the predicted probabilities for classification too.

It is wrong. Say what breaks — and say it before you turn the page.

Correct: The p(1-p) factor kills the gradient on confident mistakes, so learning stalls precisely on the hard examples the model most needs to fix.

Match the loss to the output: cross-entropy for classification, MSE for regression.

Why: The p(1-p) factor kills the gradient on confident mistakes, so learning stalls precisely on the hard examples the model most needs to fix. Verified: gradient -0.017 vs CE's -0.982 at a confident error.

Sort into buckets

Some of these hold up and some are the exact mistakes this lesson is built to prevent. Sort them.

Holds up

A model outputs a prediction f; the truth is y. A loss L(y, f) collapses the mismatch into a single non-negative number - 0 means perfect, bigger means worse.; One regression example and one classification example thread the whole deck, so every slide compounds. Regression: three targets and three model predictions.; Mean squared error takes each residual yᵢ - fᵢ, squares it, and averages. Squaring makes every miss positive and punishes big misses hardest.

Breaks

MAE is a simple linear-looking function of the error, so its gradient is just ±1/n everywhere, including when the prediction is exactly right.; A loss is a loss - MSE minimizes squared error, so just apply sigmoid + MSE to the predicted probabilities for classification too.

sound

These are stated as this lesson states them — each one survives the edge cases Lesson 17: Loss Functions & Their Gradients puts it through.

flawed

Each of these is lifted from a trap in this deck: reasonable-sounding, and wrong in a way that only shows up once you rely on it.

Discussion prompt

Explain Hinge: only punish inside the margin by analogy to something with no Machine Learning in it at all — a queue, a recipe, a map, a bank balance, whatever fits. Then say where your analogy breaks.

Hint: An analogy that never breaks is not an analogy, it is the same idea wearing a hat. Find the seam — that is the part that is actually new.

Put in order

Put the moves of Hinge gradient - three regimes into the order they have to happen.

1. If y·f > 1: L = 0, gradient 0
2. If y·f < 1: L = 1 - y·f, gradient -y
3. At y·f = 1 exactly: subgradient in [-y, 0]

Why: These are the moves of the worked example in the order it makes them, and each one is set up by the one before it. The point is safely beyond the margin - already correct with room to spare - so it contributes no push at all.

Annotate

From Hinge gradient - three regimes — read this one piece at a time. What is each part doing?

On: \( \frac{\partial L}{\partial f} = \begin{cases} 0 & y f > 1 \\ -y & y f < 1 \end{cases} \)

• The point is safely beyond the margin - already correct with room to spare - so it contributes no push at all.
• Inside the margin the loss is linear with slope -y in f. Descent (opposite -y) moves f in the +y direction, pushing the point toward the correct side.
• Same corner story as MAE - the two slopes (0 and -y) disagree at the kink, so any value between them is a valid subgradient. Optimizers just pick one.

Predict first

Evaluate loss and gradient at three points. Self-contained:

Commit before you compute: what does Hinge on three points come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: (y=+1, f=1.5): y·f = 1.5 > 1 → loss 0, grad 0

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. Correct and beyond the margin. The SVM ignores it entirely - only support vectors near the boundary matter.

Discussion prompt

Loss → gradient cheat sheet works on the cases you have just seen. Push it to the edge: what is the most degenerate input it still handles — empty, zero, one item, everything equal — and what is the first case where it stops being true? Name the case, not just "it breaks".

Hint: Try the smallest legal input, then the largest, then the one where two things collide. Methods are specified at their edges; the middle takes care of itself.

Eliminate the wrong options

For softmax + cross-entropy, the gradient of the loss with respect to the pre-softmax logits z is:

3 of these 4 are wrong. Strike them one at a time, and say what rules each one out before you strike the next. The survivor is the answer.

• A. p - y (predicted probabilities minus the one-hot target)
• B. y - p
• C. p(1 - p)
• D. -log p

Survives elimination: A

Why: In the chain rule dL/dz = Jᵀ(dL/dp), the softmax Jacobian's pₖ cancels cross-entropy's -yₖ/pₖ, leaving p - y. Verified: z = [1,2,3], true class 2 gives [0.0900, 0.2447, -0.3348].

Check your understanding

For softmax + cross-entropy, the gradient of the loss with respect to the pre-softmax logits z is:

• A. p - y (predicted probabilities minus the one-hot target) (correct)
• B. y - p
• C. p(1 - p)
• D. -log p

Answer: A

Why: In the chain rule dL/dz = Jᵀ(dL/dp), the softmax Jacobian's pₖ cancels cross-entropy's -yₖ/pₖ, leaving p - y. Verified: z = [1,2,3], true class 2 gives [0.0900, 0.2447, -0.3348].

Why B tempts people

Sign flipped. With L = -Σ y log p the gradient is p - y; y - p would point uphill and make gradient descent ASCEND the loss.

Why C tempts people

p(1 - p) is the sigmoid/softmax-diagonal DERIVATIVE. It appears (uncancelled) when you wrongly pair sigmoid with MSE - it is not in the clean CE result.

Why D tempts people

-log p is the loss VALUE for the true class, not its gradient with respect to the logits.

Predict first

Why is sigmoid + MSE a poor loss for classification?

Answer it in your own words, now, with nothing to choose from. The options are on the next slide — and picking the right one off a list is an easier skill than producing it.

Correct: Its gradient (p - y)·p(1 - p) vanishes when the model is confidently wrong

Why: The chain rule adds a p(1 - p) factor. For a confident mistake (p ≈ 0, y = 1) that factor ≈ 0, so the gradient ≈ 0 - learning stalls exactly where it matters most (-0.017 vs cross-entropy's -0.982).

Check your understanding

Why is sigmoid + MSE a poor loss for classification?

• A. Its gradient (p - y)·p(1 - p) vanishes when the model is confidently wrong (correct)
• B. MSE cannot be computed on probabilities
• C. MSE has no global minimum
• D. MSE always overfits the training data

Answer: A

Why: The chain rule adds a p(1 - p) factor. For a confident mistake (p ≈ 0, y = 1) that factor ≈ 0, so the gradient ≈ 0 - learning stalls exactly where it matters most (-0.017 vs cross-entropy's -0.982).

Why B tempts people

MSE is perfectly computable on probabilities - the issue is the gradient SHAPE (a vanishing multiplier), not whether the value exists.

Why C tempts people

MSE is convex in the prediction and does have a minimum; the failure is a vanishing gradient en route, not a missing minimum.

Why D tempts people

Overfitting is unrelated - the failure is an optimization one (no learning signal on hard cases), not a generalization one.

Predict first

For hinge loss max(0, 1 - y·f), a correctly classified point with y·f = 1.5 contributes what gradient ∂L/∂f?

Answer it in your own words, now, with nothing to choose from. The options are on the next slide — and picking the right one off a list is an easier skill than producing it.

Correct: 0 - it is beyond the margin

Why: Since y·f = 1.5 > 1, the max selects 0 and the loss is flat there, so the gradient is 0. The SVM only pushes on points inside the margin (y·f < 1) - the support vectors.

Check your understanding

For hinge loss max(0, 1 - y·f), a correctly classified point with y·f = 1.5 contributes what gradient ∂L/∂f?

• A. 0 - it is beyond the margin (correct)
• B. -y
• C. +y
• D. 1.5

Answer: A

Why: Since y·f = 1.5 > 1, the max selects 0 and the loss is flat there, so the gradient is 0. The SVM only pushes on points inside the margin (y·f < 1) - the support vectors.

Why B tempts people

-y is the gradient INSIDE the margin (y·f < 1). This point is safely outside, so it contributes nothing.

Why C tempts people

+y is never the hinge gradient in f; the active-region gradient is -y, and here it is 0 anyway.

Why D tempts people

1.5 is the signed margin value y·f, not a gradient. The gradient is 0 because the loss is flat beyond the margin.

Eliminate the wrong options

Where is MAE = (1/n)Σ|yᵢ - fᵢ| non-differentiable, and what do we use there?

3 of these 4 are wrong. Strike them one at a time, and say what rules each one out before you strike the next. The survivor is the answer.

• A. At any point where fᵢ = yᵢ; use a subgradient in [-1/n, 1/n]
• B. Nowhere - MAE is differentiable everywhere with gradient ±1/n
• C. At its minimum only, where we use the second derivative
• D. Everywhere - MAE has no usable gradient at all

Survives elimination: A

Why: |yᵢ - fᵢ| has a corner at fᵢ = yᵢ: the slope jumps from -1/n to +1/n, so no single derivative exists. Optimizers use any subgradient in [-1/n, 1/n] (numpy's sign(0) = 0 picks the middle).

Check your understanding

Where is MAE = (1/n)Σ|yᵢ - fᵢ| non-differentiable, and what do we use there?

• A. At any point where fᵢ = yᵢ; use a subgradient in [-1/n, 1/n] (correct)
• B. Nowhere - MAE is differentiable everywhere with gradient ±1/n
• C. At its minimum only, where we use the second derivative
• D. Everywhere - MAE has no usable gradient at all

Answer: A

Why: |yᵢ - fᵢ| has a corner at fᵢ = yᵢ: the slope jumps from -1/n to +1/n, so no single derivative exists. Optimizers use any subgradient in [-1/n, 1/n] (numpy's sign(0) = 0 picks the middle).

Why B tempts people

This is the trap: it misses the kink. At fᵢ = yᵢ the left slope (-1/n) and right slope (+1/n) disagree, so the derivative is undefined there.

Why C tempts people

The non-differentiable points are the kinks (fᵢ = yᵢ), not 'the minimum', and the fix is a subgradient, not a second derivative.

Why D tempts people

MAE is differentiable everywhere EXCEPT the kinks; away from fᵢ = yᵢ its gradient is a perfectly usable ±1/n.

Step through it

Step through Milestone 2 - softmax + CE → p - y one row at a time. What is driving the change, and what would the row after the last one be?

1. Step 1: class is 0
2. Step 2: class is 1
3. Step 3: class is 2 (true)

Comparison matrix

From Milestone 3 - verify against torch: every row here is a choice with a cost. Fill the dL/dz class 2 column, then say which row you would actually pick and what you give up for it.

Comparison matrix

From The full program: refill the value column from what you know. The rest of the table is as it appeared.

Discussion prompt

Stretch (homework): re-derive dL/dz = p - y for softmax+CE from the Jacobian by hand, then prove sigmoid+BCE gives the same. These exact gradients are the seed of Lesson 18's full backward pass.

That is stated as though it always holds. Do one of two things: produce a case where it fails, or say precisely what rules such a case out. "It just does" is not on the menu.

Hint: Hunt at the extremes first — zero, one, negative, empty, equal. If every extreme survives, the reason they survive is the proof.

Draw it

One page, no notation unless you need it: draw how these connect — What a loss even is · MSE - mean squared error · MAE - mean absolute error · Cross-entropy & softmax · The elegant dL/dz = p - y · Why MSE fails for classification. Put an arrow wherever one of them is what makes another possible, and label the arrow with why.