Lesson 8: Maximum Likelihood Estimation

Discussion prompt

Before we open Lesson 8: Maximum Likelihood Estimation: without looking back, what was the main idea of Linear Systems & the Normal Equations, and what could you do by the end of it that you could not do before?

Hint: One sentence for the idea, one for the skill. If the second one is blank, that is the part to revisit.

Discussion prompt

You flip a coin 10 times and see 7 heads. Nobody told you the coin's bias p. But your gut already leans: this looks like a coin that favors heads.

That is stated as though it always holds. Do one of two things: produce a case where it fails, or say precisely what rules such a case out. "It just does" is not on the menu.

Hint: Hunt at the extremes first — zero, one, negative, empty, equal. If every extreme survives, the reason they survive is the proof.

Discussion prompt

Explain The likelihood function by analogy to something with no Machine Learning in it at all — a queue, a recipe, a map, a bank balance, whatever fits. Then say where your analogy breaks.

Hint: An analogy that never breaks is not an analogy, it is the same idea wearing a hat. Find the seam — that is the part that is actually new.

Discussion prompt

Explain Likelihood is not probability of θ to a student a year behind you. No notation, no jargon they have not met — and it still has to be true.

Hint: If your explanation needs a symbol they have never seen, you are describing the notation rather than the idea.

Predict first

Let's actually evaluate the likelihood for the coin. With k = 7 heads in n = 10, L(p) = C(10,7)·pᵏ(1−p)ⁿ⁻ᵏ. Compute it for three candidate p and read off the winner:

Commit before you compute: what does Score three coins on the 7-heads data come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: Peak likelihood is at p = 0.7 — the empirical head-fraction

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. 0.7 assigns probability 0.266828 to the observed 7-of-10, beating 0.5 (0.117188) and 0.9 (0.057396).

Comparison matrix

From Score three coins on the 7-heads data: refill the L(p) = C(10,7)·p⁷(1−p)³ column from what you know. The rest of the table is as it appeared.

Discussion prompt

The product-vs-sum problem is not hypothetical. Take 2000 i.i.d. Gaussian densities and multiply them versus summing their logs — the raw product collapses to 0.0:

What do you need to know — or decide — before the first line can be written? List everything the problem has to hand you.

Hint: Anything you would have to invent to get started is a thing the problem must supply.

Comparison matrix

From Watch the product underflow: every row here is a choice with a cost. Fill the printed value column, then say which row you would actually pick and what you give up for it.

Predict first

The table runs: 3.0 | −20.8967 | far left · 4.0 | −17.8967 | climbing · 5.0 | −16.8967 | PEAK = sample mean · 6.0 | −17.8967 | descending

In The log-likelihood peaks at the same place, given the rows so far: what is the next one — the row where μ is 7.0?

Correct: 7.0 | −20.8967 | far right

Why: The relationship between the columns, not the individual numbers, is what generates the next row. The log-likelihood is a downward parabola in μ: −16.8967 at μ=5 beats −17.8967 at μ=4 and 6, and −20.8967 at μ=3 and 7.

Sort into buckets

Sort these by ℓ(μ) at σ²=4, from memory, without looking back at The log-likelihood peaks at the same place. Telling them apart on the spot is the skill; the table is only where the answer happens to be written down.

−20.8967

3.0; 7.0

−17.8967

4.0; 6.0

−16.8967

5.0

g1

ℓ(μ) at σ²=4 is "−20.8967" for 3.0, 7.0 — that is what the table on "The log-likelihood peaks at the same…" records, and it is the single property separating this group from the rest.

g2

ℓ(μ) at σ²=4 is "−17.8967" for 4.0, 6.0 — that is what the table on "The log-likelihood peaks at the same…" records, and it is the single property separating this group from the rest.

g3

ℓ(μ) at σ²=4 is "−16.8967" for 5.0 — that is what the table on "The log-likelihood peaks at the same…" records, and it is the single property separating this group from the rest.

Predict first

A student writes this, and it looks reasonable:

log transforms the function and shrinks its value, so the maximizing θ must shift too.

It is wrong. Say what breaks — and say it before you turn the page.

Correct: Confuses changing the HEIGHT of the peak (log does shrink it, often to a negative number) with changing its LOCATION (log never touches that).

log is strictly increasing, so it preserves the location of every maximum.

Why: Confuses changing the HEIGHT of the peak (log does shrink it, often to a negative number) with changing its LOCATION (log never touches that).

Annotate

From Write the Gaussian log-likelihood — read this one piece at a time. What is each part doing?

On: \( \log p(x_i;\mu,\sigma^2) = -\tfrac{1}{2}\log(2\pi\sigma^2) - \frac{(x_i-\mu)^2}{2\sigma^2} \)

• log turns the exp into its exponent and the (2πσ²)^(−1/2) into −½log(2πσ²).
• By the log trick, ℓ is the sum of the per-point logs. The constant term repeats n times.

Put in order

Put the moves of Differentiate in μ — step by step into the order they have to happen.

1. Only the sum-of-squares term contains μ
2. Differentiate Σ(xᵢ−μ)² with the chain rule
3. The two minus signs and the 2 combine into +1/σ²

Why: These are the moves of the worked example in the order it makes them, and each one is set up by the one before it. The −(n/2)log(2πσ²) term has no μ in it, so its μ-derivative is 0.

Draw it

Draw what Differentiate in μ — step by step just did — the shape of it, not the line-by-line working. One picture, labels only where you need them. Then check it against the steps: anything you could not draw is a step you followed rather than understood.

Discussion prompt

Set ∂ℓ/∂μ = 0 → the sample mean — before any calculation: what is the plan? Name the moves in order, in plain English, without doing the arithmetic.

Hint: It starts with: At the maximum the derivative is zero

Fill in the blanks

From Differentiate in σ² — step by step — finish the line. Write what belongs on the right of the equals sign before you look.

\frac-\frac{n}{2v} + \frac{1}{2v^2}\sum_i (x_i-\mu)^2___ = ___

Why: Producing the right-hand side unprompted is the difference between recognising this line and being able to use it. −(1/2v)·S = −(S/2)·v⁻¹, and d/dv(v⁻¹) = −v⁻², so the derivative is +(S/2)v⁻² = S/(2v²).

Discussion prompt

Work backwards. The example finished here:

Term 2: d/dv [−(1/2v)·S] = +S/(2v²), where S = Σ(xᵢ−μ)²

What was it asked to do, and what must it have been given? Reconstruct the problem from its answer.

Hint: Every quantity in the result had to enter somewhere. Account for each one.

Discussion prompt

Set ∂ℓ/∂σ² = 0 → the 1/n variance — before any calculation: what is the plan? Name the moves in order, in plain English, without doing the arithmetic.

Hint: It starts with: Set to zero and multiply through by 2v²

Draw it

Translate both ways. First write the expression above as a sentence with no symbols in it at all. Then cover it, and write your sentence back as notation. If the two versions disagree, the disagreement is the thing to fix.

Step through it

Step through Full trace of the variance sum one row at a time. What is driving the change, and what would the row after the last one be?

1. Step 1: xᵢ is 2
2. Step 2: xᵢ is 4
3. Step 3: xᵢ is 4
4. Step 4: xᵢ is 4
5. Step 5: xᵢ is 5
6. Step 6: xᵢ is 5
7. Step 7: xᵢ is 7
8. Step 8: xᵢ is 9

Predict first

A student writes this, and it looks reasonable:

Variance 'should' use the unbiased estimator, so divide the sum of squares by n−1.

It is wrong. Say what breaks — and say it before you turn the page.

Correct: That is Bessel's UNBIASED sample variance — a different estimator, deliberately corrected for bias.

The MLE divides by n. It is biased low, and that is the honest price of reusing the data to estimate μ.

Why: That is Bessel's UNBIASED sample variance — a different estimator, deliberately corrected for bias.

Discussion prompt

The rule this trap just fixed:

The MLE divides by n. It is biased low, and that is the honest price of reusing the data to estimate μ.

Now break it on purpose. Build a case that violates it and follow the consequences until something visibly fails. Where does the failure first show up — and would you have noticed it if you had not been looking?

Hint: The dangerous rules are the ones whose violation still produces an answer. If yours fails loudly, try to find one that fails quietly.

Predict first

np.var defaults to ddof=0 — the 1/n MLE. Pass ddof=1 for the 1/(n−1) unbiased version. Both, side by side:

Commit before you compute: what does Confirm both variances in NumPy come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: np.var(data) = 4.0 matches our by-hand MLE; ddof=1 = 4.571

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. The default np.var IS the MLE. The Bessel-corrected 32/7 = 4.5714 only appears when you ask for ddof=1.

Put in order

Put the moves of Derive the Bernoulli MLE into the order they have to happen.

1. Log-likelihood in p
2. Differentiate: dℓ/dp = k/p − (n−k)/(1−p)
3. Set to zero and cross-multiply

Why: These are the moves of the worked example in the order it makes them, and each one is set up by the one before it. Σ[xᵢ log p + (1−xᵢ)log(1−p)] collapses because Σxᵢ = k and Σ(1−xᵢ) = n−k.

Fill in the blanks

From Bernoulli MLE in code — one line has had its right-hand side removed. Put it back.

import numpy as np
b = np.array([1,0,1,1,0,1,1,0,1,1])
k = b.sum()
n = len(b)
p_mle = b.mean() # = k/n
print(k, n, p_mle)

Why: n is what everything below it consumes, so the wrong expression here fails later and somewhere else. The mean of a 0/1 array IS the success fraction.

Sort into buckets

These are the pieces of Lesson 8: Maximum Likelihood Estimation, out of order. Put each one back under the part of the lesson it belongs to.

The likelihood principle

A coin told you something; The likelihood function; Likelihood is not probability of θ

MLE for the Gaussian

Our running dataset; Write the Gaussian log-likelihood; Differentiate in μ — step by step

MLE for the Bernoulli

The coin, finished; Derive the Bernoulli MLE; Bernoulli MLE in code

s1

The likelihood principle is where Lesson 8: Maximum Likelihood Estimation puts A coin told you something, The likelihood function, Likelihood is not probability of θ. Knowing which part of the lesson a problem belongs to is most of knowing which method to reach for.

s2

MLE for the Gaussian is where Lesson 8: Maximum Likelihood Estimation puts Our running dataset, Write the Gaussian log-likelihood, Differentiate in μ — step by step. Knowing which part of the lesson a problem belongs to is most of knowing which method to reach for.

s3

MLE for the Bernoulli is where Lesson 8: Maximum Likelihood Estimation puts The coin, finished, Derive the Bernoulli MLE, Bernoulli MLE in code. Knowing which part of the lesson a problem belongs to is most of knowing which method to reach for.

Put in order

These are the steps of The MLE recipe, scrambled. Put them back in order before the next slide shows you.

1. Write the model p(x; θ) for one data point
2. Log & sum: ℓ(θ) = Σ log p(xᵢ; θ) — the log trick turns the product into a sum
3. Differentiate ℓ with respect to each parameter
4. Set to zero and solve for the parameter (check it is a max, not a min)
5. Plug in the data summaries (Σxᵢ, k, Σxᵢ², …) to get a number

Why: This is the order the recipe itself gives. Recalling the sequence without the slide in front of you is the difference between recognising the method and being able to run it — most of what goes wrong in practice is a step done out of turn.

Fill in the blanks

From Poisson and Exponential fall out too — one line has had its right-hand side removed. Put it back.

import numpy as np
pois = np.array([2,3,1,4,2,3])
ex = np.array([0.5,1.5,2.0,1.0])
print('Poisson lambda =', pois.mean()) # MLE = sample mean
print('Exp lambda =', 1/ex.mean()) # MLE = 1/sample mean

Why: ex is what everything below it consumes, so the wrong expression here fails later and somewhere else. d/dλ Σ(xᵢ log λ − λ) = Σxᵢ/λ − n = 0 ⇒ λ = Σxᵢ/n.

Annotate

Walk the callouts on Poisson and Exponential fall out too. Each one is a place this is easy to get subtly wrong.

• d/dλ Σ(xᵢ log λ − λ) = Σxᵢ/λ − n = 0 ⇒ λ = Σxᵢ/n. For [2,3,1,4,2,3], that is 15/6 = 2.5.
• d/dλ = n/λ − Σxᵢ = 0 ⇒ λ = n/Σxᵢ = 1/x̄. For mean 1.25, λ = 0.8.

Eliminate the wrong options

Maximizing ℓ(θ) = Σ log p(xᵢ;θ) gives the same θ as maximizing ∏ p(xᵢ;θ) because…

3 of these 4 are wrong. Strike them one at a time, and say what rules each one out before you strike the next. The survivor is the answer.

• A. log is strictly increasing, so it preserves the location of the maximum
• B. log(∏) happens to equal ∏ log for probabilities
• C. the likelihood and log-likelihood have the same numerical value
• D. taking log removes the dependence on θ

Survives elimination: A

Why: A monotonically increasing transform changes the height of the peak but never its location, so the argmax is unchanged. As a bonus, log turns the product into a sum, which is why we use it.

Check your understanding

Maximizing ℓ(θ) = Σ log p(xᵢ;θ) gives the same θ as maximizing ∏ p(xᵢ;θ) because…

• A. log is strictly increasing, so it preserves the location of the maximum (correct)
• B. log(∏) happens to equal ∏ log for probabilities
• C. the likelihood and log-likelihood have the same numerical value
• D. taking log removes the dependence on θ

Answer: A

Why: A monotonically increasing transform changes the height of the peak but never its location, so the argmax is unchanged. As a bonus, log turns the product into a sum, which is why we use it.

Why B tempts people

log(∏) = Σ log, not ∏ log. The product becomes a SUM of logs — that's the useful identity, stated wrong here.

Why C tempts people

The values differ — log shrinks them, often making them negative (our peak ℓ was −16.8967). What's preserved is the LOCATION of the max, not the value.

Why D tempts people

The log-likelihood still depends on θ — if it didn't, there would be nothing left to maximize.

Predict first

For Gaussian data, the maximum-likelihood estimate of the variance divides the sum of squared deviations by…

Answer it in your own words, now, with nothing to choose from. The options are on the next slide — and picking the right one off a list is an easier skill than producing it.

Correct: n (and is biased low)

Why: Setting ∂ℓ/∂σ² = 0 yields σ²_MLE = (1/n)Σ(xᵢ−μ)². It is biased low because the same data estimates both μ and σ. On our data that is 32/8 = 4.0.

Check your understanding

For Gaussian data, the maximum-likelihood estimate of the variance divides the sum of squared deviations by…

• A. n (and is biased low) (correct)
• B. n−1 (and is unbiased)
• C. n+1
• D. √n

Answer: A

Why: Setting ∂ℓ/∂σ² = 0 yields σ²_MLE = (1/n)Σ(xᵢ−μ)². It is biased low because the same data estimates both μ and σ. On our data that is 32/8 = 4.0.

Why B tempts people

1/(n−1) is the unbiased (Bessel-corrected) sample variance — 32/7 ≈ 4.571 here — a deliberately bias-corrected estimator, not the MLE.

Why C tempts people

n+1 minimizes the mean squared error of the variance estimate in some settings, but it is neither the MLE nor the unbiased estimator.

Why D tempts people

√n appears in the standard ERROR of the mean, not in the variance estimator's denominator.

Discussion prompt

Read the picture before the words. What is this showing, and what is the one thing it is built to make obvious? Commit to an answer, then read on.

Hint: Name the parts, then say what changes between them — and if nothing changes, say what is being held still.

Discussion prompt

From Gaussian likelihood to MSE — before any calculation: what is the plan? Name the moves in order, in plain English, without doing the arithmetic.

Hint: It starts with: Negative log-likelihood of the targets

Draw it

Draw what From Gaussian likelihood to MSE just did — the shape of it, not the line-by-line working. One picture, labels only where you need them. Then check it against the steps: anything you could not draw is a step you followed rather than understood.

Fill in the blanks

From The tie-back to Lesson 7, in code — one line has had its right-hand side removed. Put it back.

import numpy as np
x = np.array([1.,2.,3.,4.,5.])
y = np.array([2.,4.,5.,4.,5.])
X = np.c_[np.ones(5), x]
w = np.linalg.solve(X.T @ X, X.T @ y)
sse = np.sum((y - X @ w)**2)
print(w, sse, sse/5)

Why: X is what everything below it consumes, so the wrong expression here fails later and somewhere else. Minimizing MSE gave the same w = [2.2, 0.6] as Lesson 7, and the noise-variance MLE is just the mean squared residual, 0.48.

Predict first

The table runs: (1, 0.9) | −log(0.9) | 0.1054 · (0, 0.2) | −log(0.8) | 0.2231 · (1, 0.7) | −log(0.7) | 0.3567

In BCE equals the negative log-likelihood, given the rows so far: what is the next one — the row where point (y, p) is mean of the three?

Correct: mean of the three | = F.binary_cross_entropy | 0.2284

Why: The relationship between the columns, not the individual numbers, is what generates the next row. y=1 point contributes −log(0.9)=0.1054; y=0 point contributes −log(1−0.2)=−log(0.8)=0.2231; y=1 point contributes −log(0.7)=0.3567.

Comparison matrix

From BCE equals the negative log-likelihood: refill the −log-likelihood term column from what you know. The rest of the table is as it appeared.

Predict first

A student writes this, and it looks reasonable:

MLE maximizes likelihood, and 'the loss is the likelihood', so training should maximize the cross-entropy.

It is wrong. Say what breaks — and say it before you turn the page.

Correct: This silently drops the negative sign.

The loss is the negative log-likelihood, so maximizing likelihood = minimizing loss.

Why: This silently drops the negative sign. Cross-entropy is the NEGATED log-likelihood, so climbing it drives the likelihood DOWN.

Predict first

Minimizing mean squared error is equivalent to maximum likelihood under which assumption?

Commit to an answer, then rate it — certain, fairly sure, or guessing — and write the rating down before you turn the page.

Correct: The target noise is Gaussian with constant variance

Why: With y = f(x) + ε and ε ~ N(0,σ²), the negative log-likelihood reduces to (1/2σ²)Σ(y−f)² plus constants — exactly MSE.

The rating matters as much as the answer: confident-and-wrong is the combination that survives revision, because nothing about it feels like it needs revisiting.

Check your understanding

Minimizing mean squared error is equivalent to maximum likelihood under which assumption?

• A. The target noise is Gaussian with constant variance (correct)
• B. The target noise is Laplace-distributed
• C. The labels are Bernoulli
• D. No assumption — MSE is always the likelihood

Answer: A

Why: With y = f(x) + ε and ε ~ N(0,σ²), the negative log-likelihood reduces to (1/2σ²)Σ(y−f)² plus constants — exactly MSE.

Why B tempts people

A Laplace noise model gives the MEAN ABSOLUTE error (L1 loss), not squared error — the |·| comes from e^{−|·|}.

Why C tempts people

Bernoulli labels give cross-entropy, the classification loss — not MSE for regression.

Why D tempts people

MSE corresponds to a SPECIFIC model (Gaussian noise). Under other noise models the MLE loss is different.

Predict first

Training a classifier by minimizing binary cross-entropy is equivalent to…

Answer it in your own words, now, with nothing to choose from. The options are on the next slide — and picking the right one off a list is an easier skill than producing it.

Correct: maximizing the Bernoulli log-likelihood of the labels

Why: BCE is the averaged NEGATIVE Bernoulli log-likelihood, so min BCE = max ℓ. Our numeric check gave the same 0.2284 for the hand NLL and torch's BCE.

Check your understanding

Training a classifier by minimizing binary cross-entropy is equivalent to…

• A. maximizing the Bernoulli log-likelihood of the labels (correct)
• B. maximizing the cross-entropy of the labels
• C. minimizing the log-likelihood of the labels
• D. minimizing the sum of squared probabilities

Answer: A

Why: BCE is the averaged NEGATIVE Bernoulli log-likelihood, so min BCE = max ℓ. Our numeric check gave the same 0.2284 for the hand NLL and torch's BCE.

Why B tempts people

You minimize cross-entropy, not maximize it — maximizing it (gradient ascent) is the Part-4 trap that drives predictions the wrong way.

Why C tempts people

min(−ℓ) is max(ℓ), not min(ℓ). Minimizing the log-likelihood itself would make the data less probable — backwards.

Why D tempts people

Squared probabilities are the Brier score, a different loss; BCE comes from the Bernoulli LOG-likelihood, not squared terms.

Put in order

Put the moves of A Gaussian prior IS the ridge penalty into the order they have to happen.

1. Put a Gaussian prior on the weights: w ~ N(0, τ²I)
2. MAP objective = log-likelihood + log-prior
3. Identify λ = σ²/τ² — this is ridge from Lesson 7

Why: These are the moves of the worked example in the order it makes them, and each one is set up by the one before it. A belief that weights are small and centered at 0 — the same belief ridge encodes.

Annotate

From A Gaussian prior IS the ridge penalty — read this one piece at a time. What is each part doing?

On: \( -\big[\ell + \log p(w)\big] \;\propto\; \lVert Xw - y\rVert^2 + \frac{\sigma^2}{\tau^2}\lVert w\rVert^2 \)

• A belief that weights are small and centered at 0 — the same belief ridge encodes.
• Add the Gaussian-noise log-likelihood (→ −½σ⁻²·SSE) to the log-prior (→ −½τ⁻²‖w‖²), then negate to get a loss to minimize.
• The log-prior contributed exactly λ‖w‖². Ridge's penalty was never ad hoc — it is a Gaussian prior on the weights. A Laplace prior would give L1 instead.

Discussion prompt

Explain The prior shrinks the estimate to a student a year behind you. No notation, no jargon they have not met — and it still has to be true.

Hint: If your explanation needs a symbol they have never seen, you are describing the notation rather than the idea.

Discussion prompt

Estimate the mean of our 8-point data (x̄ = 5, σ² = 4) under a N(0, τ²) prior. Watch a tighter prior pull the estimate down from 5 toward 0:

What do you need to know — or decide — before the first line can be written? List everything the problem has to hand you.

Hint: Anything you would have to invent to get started is a thing the problem must supply.

Discussion prompt

Work backwards. The example finished here:

τ²=∞ → 5.0 (pure MLE); τ²=4 → 4.4444; τ²=1 → 3.3333

What was it asked to do, and what must it have been given? Reconstruct the problem from its answer.

Hint: Every quantity in the result had to enter somewhere. Account for each one.

Predict first

A student writes this, and it looks reasonable:

MAP is Bayesian and MLE is frequentist, so MAP must need a totally separate machinery.

It is wrong. Say what breaks — and say it before you turn the page.

Correct: Misses that after taking the log, the prior is just ANOTHER additive term on the very same objective.

MAP is MLE plus one extra term: the log-prior. Same optimization, one added regularizer.

Why: Misses that after taking the log, the prior is just ANOTHER additive term on the very same objective.

Sort into buckets

Some of these hold up and some are the exact mistakes this lesson is built to prevent. Sort them.

Holds up

You flip a coin 10 times and see 7 heads. Nobody told you the coin's bias p. But your gut already leans: this looks like a coin that favors heads.; A subtle point that trips people up: L(θ) is not the probability that θ is correct. It is the probability the model assigns to the data, read as a function of θ.; Maximum likelihood estimation picks the parameter that maximizes that function — the θ that makes exactly this dataset as probable as possible:

Breaks

log transforms the function and shrinks its value, so the maximizing θ must shift too.; Variance 'should' use the unbiased estimator, so divide the sum of squares by n−1.

sound

These are stated as this lesson states them — each one survives the edge cases Lesson 8: Maximum Likelihood Estimation puts it through.

flawed

Each of these is lifted from a trap in this deck: reasonable-sounding, and wrong in a way that only shows up once you rely on it.

Put in order

These are the steps of Deriving a loss from a model, scrambled. Put them back in order before the next slide shows you.

1. Write the probabilistic model p(x; θ) or p(y | x; θ)
2. Form the log-likelihood ℓ(θ) = Σ log p
3. Negate it → that's your loss L = −ℓ (Gaussian noise → MSE, Bernoulli → BCE)
4. Add −log p(θ) if you have a prior → the regularizer (Gaussian → L2, Laplace → L1)
5. Minimize L by gradient descent — the Lesson 6 loop

Why: This is the order the recipe itself gives. Recalling the sequence without the slide in front of you is the difference between recognising the method and being able to run it — most of what goes wrong in practice is a step done out of turn.

Discussion prompt

Deriving a loss from a model works on the cases you have just seen. Push it to the edge: what is the most degenerate input it still handles — empty, zero, one item, everything equal — and what is the first case where it stops being true? Name the case, not just "it breaks".

Hint: Try the smallest legal input, then the largest, then the one where two things collide. Methods are specified at their edges; the middle takes care of itself.

Eliminate the wrong options

Adding an L2 (ridge) penalty λ‖w‖² to a squared-error loss is equivalent to MAP estimation with which prior on the weights?

3 of these 4 are wrong. Strike them one at a time, and say what rules each one out before you strike the next. The survivor is the answer.

• A. A zero-mean Gaussian prior, w ~ N(0, τ²I)
• B. A Laplace prior, p(w) ∝ e^{−|w|/b}
• C. A uniform prior over all weights
• D. A Bernoulli prior on each weight

Survives elimination: A

Why: log of a zero-mean Gaussian prior is −‖w‖²/(2τ²) + const, which is exactly the L2 penalty with λ = σ²/τ². A tighter prior (smaller τ²) means larger λ and stronger shrinkage.

Check your understanding

Adding an L2 (ridge) penalty λ‖w‖² to a squared-error loss is equivalent to MAP estimation with which prior on the weights?

• A. A zero-mean Gaussian prior, w ~ N(0, τ²I) (correct)
• B. A Laplace prior, p(w) ∝ e^{−|w|/b}
• C. A uniform prior over all weights
• D. A Bernoulli prior on each weight

Answer: A

Why: log of a zero-mean Gaussian prior is −‖w‖²/(2τ²) + const, which is exactly the L2 penalty with λ = σ²/τ². A tighter prior (smaller τ²) means larger λ and stronger shrinkage.

Why B tempts people

A Laplace prior gives the L1 penalty (|w|, lasso), not the squared L2 penalty — its log is −|w|/b, not −w²/(2τ²).

Why C tempts people

A uniform (flat) prior has constant log, adds nothing to the objective, and reduces MAP back to plain MLE — no penalty at all.

Why D tempts people

Bernoulli is a distribution over 0/1 outcomes, not a sensible prior over continuous real-valued weights.

Discussion prompt

Explain Project: maximum_likelihood_gaussian(data) by analogy to something with no Machine Learning in it at all — a queue, a recipe, a map, a bank balance, whatever fits. Then say where your analogy breaks.

Hint: An analogy that never breaks is not an analogy, it is the same idea wearing a hat. Find the seam — that is the part that is actually new.

Step through it

Step through Milestone 2 — the variance one row at a time. What is driving the change, and what would the row after the last one be?

1. Step 1: quantity is Σ(x−μ)²
2. Step 2: quantity is σ²_MLE = 32/8
3. Step 3: quantity is σ_MLE = √4

Predict first

Assemble the three milestones into one function and prove it agrees with NumPy in a single boolean:

Commit before you compute: what does The full function come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: Returns (5.0, 2.0); np.allclose vs np.var prints True

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. Your from-scratch MLE matches NumPy's default variance exactly.

Comparison matrix

From The full function: every row here is a choice with a cost. Fill the value column, then say which row you would actually pick and what you give up for it.

Discussion prompt

Slides closed, out loud: explain (1) why the log trick never moves the argmax, (2) why MSE and cross-entropy are both negative log-likelihoods, and (3) which prior turns MLE into ridge.

That is stated as though it always holds. Do one of two things: produce a case where it fails, or say precisely what rules such a case out. "It just does" is not on the menu.

Hint: Hunt at the extremes first — zero, one, negative, empty, equal. If every extreme survives, the reason they survive is the proof.

Comparison matrix

From One principle, every loss: refill the the loss it becomes column from what you know. The rest of the table is as it appeared.

Draw it

One page, no notation unless you need it: draw how these connect — The likelihood principle · MLE for the Gaussian · MLE for the Bernoulli · Losses are negative log-likelihoods · MAP: MLE plus a prior · Your turn: build the MLE. Put an arrow wherever one of them is what makes another possible, and label the arrow with why.