Lesson 41: Logistic Regression

Discussion prompt

Before we open Lesson 41: Logistic Regression: without looking back, what was the main idea of Linear Regression & the PyTorch Training Loop, and what could you do by the end of it that you could not do before?

Hint: One sentence for the idea, one for the skill. If the second one is blank, that is the part to revisit.

Discussion prompt

Logistic regression models the probability of class 1 as a sigmoid of a linear score. The key constraint: this must stay in (0, 1).

That is stated as though it always holds. Do one of two things: produce a case where it fails, or say precisely what rules such a case out. "It just does" is not on the menu.

Hint: Hunt at the extremes first — zero, one, negative, empty, equal. If every extreme survives, the reason they survive is the proof.

Put in order

Put the moves of Deriving BCE from Bernoulli log-likelihood into the order they have to happen.

1. Write the Bernoulli likelihood for one example (y ∈ {0,1})
2. Take the negative log to convert product to sum
3. Note that L is convex in w (unlike MSE on probabilities)

Why: These are the moves of the worked example in the order it makes them, and each one is set up by the one before it. p(y|x) = p^y · (1-p)^(1-y) unifies both cases in one expression.

Annotate

From Deriving BCE from Bernoulli log-likelihood — read this one piece at a time. What is each part doing?

On: \( \mathcal{L}(w) = \prod_{i=1}^{n} \hat{p}_i^{y_i}(1-\hat{p}_i)^{1-y_i} \)

• p(y|x) = p^y · (1-p)^(1-y) unifies both cases in one expression.
• log turns products into sums; negation converts maximizing likelihood to minimizing loss.
• The sigmoid composed with log is concave in the linear score; negation makes the overall objective convex — so gradient descent finds the global minimum.

Put in order

Put the moves of Proving dL/dw = X^T(p − y)/n into the order they have to happen.

1. Differentiate L w.r.t. the linear score z = w^T x_i
2. Chain rule again: dL/dw = (dL/dz) · (dz/dw) = (p_i - y_i) · x_i
3. Finite-difference check confirms the formula exactly

Why: These are the moves of the worked example in the order it makes them, and each one is set up by the one before it. Chain rule: dL/dz = -[y/p - (1-y)/(1-p)] · dp/dz.

Annotate

From Proving dL/dw = X^T(p − y)/n — read this one piece at a time. What is each part doing?

On: \( \frac{\partial L}{\partial w} = \frac{1}{n}X^\top(\hat{p} - y) \quad\text{(verified numerically, max error } {<} 10^{-11}\text{)} \)

• Chain rule: dL/dz = -[y/p - (1-y)/(1-p)] · dp/dz. Use dp/dz = p(1-p) (sigmoid derivative).
• dz/dw = x_i; summing over all n samples and dividing by n gives the compact matrix form.
• Numerically computed gradient (central differences) matches the analytic formula to < 10^-11 on a 10-sample test.

Comparison matrix

From Synthetic binary dataset: refill the center column from what you know. The rest of the table is as it appeared.

Predict first

Implement the gradient X^T(p-y)/n directly. No autograd — you own every computation.

Commit before you compute: what does NumPy manual backprop come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: w = [-0.1206, 1.3404, 1.8292], acc = 0.8950 after 250 epochs

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. The gradient formula X^T(p-y)/n is the complete backprop for logistic regression — no library needed.

Comparison matrix

From NumPy manual backprop: every row here is a choice with a cost. Fill the w1 column, then say which row you would actually pick and what you give up for it.

Discussion prompt

Same dataset, same training loop rhythm from L40: zero_grad → forward → loss → backward → step.

What do you need to know — or decide — before the first line can be written? List everything the problem has to hand you.

Hint: Anything you would have to invent to get started is a thing the problem must supply.

Discussion prompt

Work backwards. The example finished here:

PyTorch BCELoss converges to same weights as NumPy: [1.3400, 1.8301], bias −0.1207, acc 0.8950

What was it asked to do, and what must it have been given? Reconstruct the problem from its answer.

Hint: Every quantity in the result had to enter somewhere. Account for each one.

Predict first

A student writes this, and it looks reasonable:

Use nn.Sigmoid() in the model and pass output to nn.BCELoss() — this is cleaner because the model output is already a probability.

It is wrong. Say what breaks — and say it before you turn the page.

Correct: Float32 saturates: sigmoid(50) = 1.0 to machine precision.

Use nn.Linear (raw logits) with nn.BCEWithLogitsLoss() — no sigmoid in the model.

Why: Float32 saturates: sigmoid(50) = 1.0 to machine precision. Then BCELoss computes log(1 - 1.0) = log(0.0) = −inf, which propagates to NaN gradients and a broken model.

Discussion prompt

The rule this trap just fixed:

For z = 50: log(1 + exp(-50)) ≈ 0 exactly — no saturation. The numerics stay finite across the full logit range. Same final value: 0.2310 for our extreme test, but no NaN risk.

Now break it on purpose. Build a case that violates it and follow the consequences until something visibly fails. Where does the failure first show up — and would you have noticed it if you had not been looking?

Hint: The dangerous rules are the ones whose violation still produces an answer. If yours fails loudly, try to find one that fails quietly.

Discussion prompt

Explain BCEWithLogitsLoss: the log-sum-exp trick by analogy to something with no Machine Learning in it at all — a queue, a recipe, a map, a bank balance, whatever fits. Then say where your analogy breaks.

Hint: An analogy that never breaks is not an analogy, it is the same idea wearing a hat. Find the seam — that is the part that is actually new.

Discussion prompt

Explain From binary to K-class to a student a year behind you. No notation, no jargon they have not met — and it still has to be true.

Hint: If your explanation needs a symbol they have never seen, you are describing the notation rather than the idea.

Predict first

3-class example: logits = [3.0, 1.0, −2.0], true class = 0. Compute softmax probabilities and cross-entropy loss.

Commit before you compute: what does Softmax CE by hand come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: Softmax probs: [0.8756, 0.1185, 0.0059]. CE loss = 0.1328

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. Class 0 has logit 3.0, far above the others, so it captures 87.6% of the probability mass.

Discussion prompt

Work backwards. The example finished here:

Softmax probs: [0.8756, 0.1185, 0.0059]. CE loss = 0.1328

What was it asked to do, and what must it have been given? Reconstruct the problem from its answer.

Hint: Every quantity in the result had to enter somewhere. Account for each one.

Predict first

Iris: 150 samples, 4 features, 3 classes — the canonical multi-class test. One nn.Linear(4, 3) + CrossEntropyLoss.

Commit before you compute: what does Softmax regression on Iris (3 classes) come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: Final accuracy 0.9733 on Iris; final CE loss 0.1734 — softmax regression with one linear layer

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. Iris is nearly linearly separable; a single linear layer (no hidden units) reaches 97.3% in 500 epochs — same loop as L40, new loss.

Step through it

Step through Softmax regression on Iris (3 classes) one row at a time. What is driving the change, and what would the row after the last one be?

1. Step 1: epoch is 0
2. Step 2: epoch is 100
3. Step 3: epoch is 200
4. Step 4: epoch is 300
5. Step 5: epoch is 500 (final)

Discussion prompt

Explain Decision boundary and regularization by analogy to something with no Machine Learning in it at all — a queue, a recipe, a map, a bank balance, whatever fits. Then say where your analogy breaks.

Hint: An analogy that never breaks is not an analogy, it is the same idea wearing a hat. Find the seam — that is the part that is actually new.

Predict first

A student writes this, and it looks reasonable:

Since softmax outputs probabilities, apply it in the model and pass probabilities to CrossEntropyLoss — makes the model output interpretable.

It is wrong. Say what breaks — and say it before you turn the page.

Correct: CrossEntropyLoss already applies log-softmax internally (log-sum-exp stable).

Pass raw logits to CrossEntropyLoss. Add nn.Softmax only at inference for human-readable probabilities.

Why: CrossEntropyLoss already applies log-softmax internally (log-sum-exp stable). Feeding pre-softmax probabilities into it computes softmax(softmax(z)) — double-squashing the logits, producing badly distorted gradients.

Sort into buckets

Some of these hold up and some are the exact mistakes this lesson is built to prevent. Sort them.

Holds up

Logistic regression models the probability of class 1 as a sigmoid of a linear score. The key constraint: this must stay in (0, 1).; Generate 200 points from two Gaussian clusters: class 1 centered at (1,1), class 0 at (-1,-1). Both with covariance [[1, 0.3],[0.3, 1]]. Seed 42.; The fix is computing the BCE loss directly from logits using a numerically equivalent form.

Breaks

Use nn.Sigmoid() in the model and pass output to nn.BCELoss() — this is cleaner because the model output is already a probability.; Since softmax outputs probabilities, apply it in the model and pass probabilities to CrossEntropyLoss — makes the model output interpretable.

sound

These are stated as this lesson states them — each one survives the edge cases Lesson 41: Logistic Regression puts it through.

flawed

Each of these is lifted from a trap in this deck: reasonable-sounding, and wrong in a way that only shows up once you rely on it.

Comparison matrix

From Logistic regression vs linear SVM: refill the logistic regression column from what you know. The rest of the table is as it appeared.

Put in order

These are the steps of The logistic regression recipe, scrambled. Put them back in order before the next slide shows you.

1. Loss: negative log-likelihood of Bernoulli = BCE. Gradient = X^T(p−y)/n (proved from scratch).
2. Binary: nn.Linear(d,1) + nn.BCEWithLogitsLoss() — never sigmoid → BCELoss on extreme logits
3. Multi-class: nn.Linear(d,K) + nn.CrossEntropyLoss() — softmax applied internally, pass raw logits
4. Loop: identical to L40 — zero_grad → forward → loss → backward → step
5. Regularize: weight_decay=λ ↔ L2 penalty; shrinks ||w||, smooths the decision boundary
6. Interpret: predict class via logits.argmax(); probabilities via sigmoid (binary) or softmax (multi)

Why: This is the order the recipe itself gives. Recalling the sequence without the slide in front of you is the difference between recognising the method and being able to run it — most of what goes wrong in practice is a step done out of turn.

Discussion prompt

The logistic regression recipe works on the cases you have just seen. Push it to the edge: what is the most degenerate input it still handles — empty, zero, one item, everything equal — and what is the first case where it stops being true? Name the case, not just "it breaks".

Hint: Try the smallest legal input, then the largest, then the one where two things collide. Methods are specified at their edges; the middle takes care of itself.

Eliminate the wrong options

The gradient of binary cross-entropy loss w.r.t. w is:

3 of these 4 are wrong. Strike them one at a time, and say what rules each one out before you strike the next. The survivor is the answer.

• A. X^T(p − y) / n
• B. X^T(y − p) / n
• C. X^T(p − y) · p(1−p)
• D. 2 · X^T(Xw − y) / n

Survives elimination: A

Why: The chain rule yields dL/dz_i = p_i − y_i (the sigmoid derivative cancels cleanly with the log-likelihood). Summing over samples in matrix form gives X^T(p−y)/n. Verified numerically against finite differences with max error < 10^{-11}.

Check your understanding

The gradient of binary cross-entropy loss w.r.t. w is:

• A. X^T(p − y) / n (correct)
• B. X^T(y − p) / n
• C. X^T(p − y) · p(1−p)
• D. 2 · X^T(Xw − y) / n

Answer: A

Why: The chain rule yields dL/dz_i = p_i − y_i (the sigmoid derivative cancels cleanly with the log-likelihood). Summing over samples in matrix form gives X^T(p−y)/n. Verified numerically against finite differences with max error < 10^{-11}.

Why B tempts people

y − p has the wrong sign; that would be the gradient of the negative of the loss (i.e., the log-likelihood ascent direction).

Why C tempts people

Including p(1-p) double-counts the sigmoid derivative — the chain rule already absorbed it in the dL/dz step.

Why D tempts people

That's the MSE gradient (L40), not cross-entropy. Logistic regression uses BCE, not squared error.

Predict first

Why is BCEWithLogitsLoss preferred over nn.Sigmoid() + nn.BCELoss()?

Answer it in your own words, now, with nothing to choose from. The options are on the next slide — and picking the right one off a list is an easier skill than producing it.

Correct: Float32 sigmoid saturates at large logit magnitudes, making log(1−sigmoid(z)) = log(0) = −inf; BCEWithLogitsLoss avoids this with the log-sum-exp trick

Why: sigmoid(50) = 1.0 in float32. BCELoss then computes log(0) = −inf for a class-0 sample, giving NaN gradients. BCEWithLogitsLoss reformulates as max(z,0) − yz + log(1+exp(−|z|)), which is finite for any z. Both compute the same value for moderate logits (verified: 0.2310 on our test).

Check your understanding

Why is BCEWithLogitsLoss preferred over nn.Sigmoid() + nn.BCELoss()?

• A. Float32 sigmoid saturates at large logit magnitudes, making log(1−sigmoid(z)) = log(0) = −inf; BCEWithLogitsLoss avoids this with the log-sum-exp trick (correct)
• B. BCELoss produces a non-convex loss surface, BCEWithLogitsLoss is convex
• C. BCEWithLogitsLoss trains faster because it skips the sigmoid computation
• D. BCELoss requires class labels in {−1, +1} while BCEWithLogitsLoss accepts {0, 1}

Answer: A

Why: sigmoid(50) = 1.0 in float32. BCELoss then computes log(0) = −inf for a class-0 sample, giving NaN gradients. BCEWithLogitsLoss reformulates as max(z,0) − yz + log(1+exp(−|z|)), which is finite for any z. Both compute the same value for moderate logits (verified: 0.2310 on our test).

Why B tempts people

BCE is convex in z = w^Tx regardless of whether you split the sigmoid — both formulations are convex.

Why C tempts people

BCEWithLogitsLoss still computes the sigmoid internally — the speed difference is negligible; the motivation is numerical precision.

Why D tempts people

Both BCELoss and BCEWithLogitsLoss expect labels in {0, 1} for standard use; the label convention is the same.

Eliminate the wrong options

You build a 3-class classifier: model = nn.Sequential(nn.Linear(4,3), nn.Softmax(dim=1)). You train with nn.CrossEntropyLoss()(model(x), y). What is wrong?

3 of these 4 are wrong. Strike them one at a time, and say what rules each one out before you strike the next. The survivor is the answer.

• A. CrossEntropyLoss applies softmax internally — passing pre-softmax output squashes the logits twice, distorting gradients
• B. Nothing — Softmax normalizes to probabilities and CrossEntropyLoss then computes cross-entropy correctly
• C. Softmax should be replaced by Sigmoid for multi-class
• D. The output dimension should be 1, not 3, for 3 classes

Survives elimination: A

Why: nn.CrossEntropyLoss = log-softmax + NLLLoss. If you pass probabilities (already softmax'd) it computes softmax(softmax(z)), which compresses all activations toward uniform — training appears to work but converges much more slowly and to a suboptimal solution. Always pass raw logits.

Check your understanding

You build a 3-class classifier: model = nn.Sequential(nn.Linear(4,3), nn.Softmax(dim=1)). You train with nn.CrossEntropyLoss()(model(x), y). What is wrong?

• A. CrossEntropyLoss applies softmax internally — passing pre-softmax output squashes the logits twice, distorting gradients (correct)
• B. Nothing — Softmax normalizes to probabilities and CrossEntropyLoss then computes cross-entropy correctly
• C. Softmax should be replaced by Sigmoid for multi-class
• D. The output dimension should be 1, not 3, for 3 classes

Answer: A

Why: nn.CrossEntropyLoss = log-softmax + NLLLoss. If you pass probabilities (already softmax'd) it computes softmax(softmax(z)), which compresses all activations toward uniform — training appears to work but converges much more slowly and to a suboptimal solution. Always pass raw logits.

Why B tempts people

This is the misconception — CrossEntropyLoss does not 'receive probabilities and compute cross-entropy'; it expects logits and applies log-softmax itself.

Why C tempts people

Sigmoid is for binary output (each unit independent). For mutually exclusive K-class output, softmax is correct — but it belongs inside the loss, not the model.

Why D tempts people

The output size must match K (number of classes). For 3 classes, nn.Linear(4, 3) is correct.

Discussion prompt

Implement binary logistic regression from scratch (NumPy) and in PyTorch, then extend to multi-class. Three milestones.

That is stated as though it always holds. Do one of two things: produce a case where it fails, or say precisely what rules such a case out. "It just does" is not on the menu.

Hint: Hunt at the extremes first — zero, one, negative, empty, equal. If every extreme survives, the reason they survive is the proof.

Step through it

Step through Milestone 1 — NumPy manual backprop one row at a time. What is driving the change, and what would the row after the last one be?

1. Step 1: param is w_bias
2. Step 2: param is w1
3. Step 3: param is w2
4. Step 4: param is accuracy

Comparison matrix

From Milestone 3 — Iris softmax regression: every row here is a choice with a cost. Fill the value column, then say which row you would actually pick and what you give up for it.

Comparison matrix

From The full program: refill the result column from what you know. The rest of the table is as it appeared.

Draw it

One page, no notation unless you need it: draw how these connect — Binary cross-entropy from scratch · NumPy vs PyTorch implementation · Softmax regression and multi-class · Your turn: implement it. Put an arrow wherever one of them is what makes another possible, and label the arrow with why.