Lesson 12: MAP, Priors & Regularization

Discussion prompt

Before we open Lesson 12: MAP, Priors & Regularization: without looking back, what was the main idea of Autoencoders — Standard, Denoising & Sparse, and what could you do by the end of it that you could not do before?

Hint: One sentence for the idea, one for the skill. If the second one is blank, that is the part to revisit.

Comparison matrix

From The running example: fit a line, but small data: refill the hours x column from what you know. The rest of the table is as it appeared.

Discussion prompt

In Lesson 8 the maximum-likelihood estimate picked the θ that makes the observed data most probable. For Gaussian noise on a linear model, that is exactly least squares.

That is stated as though it always holds. Do one of two things: produce a case where it fails, or say precisely what rules such a case out. "It just does" is not on the menu.

Hint: Hunt at the extremes first — zero, one, negative, empty, equal. If every extreme survives, the reason they survive is the proof.

Discussion prompt

Explain Bayes' rule, for parameters by analogy to something with no Machine Learning in it at all — a queue, a recipe, a map, a bank balance, whatever fits. Then say where your analogy breaks.

Hint: An analogy that never breaks is not an analogy, it is the same idea wearing a hat. Find the seam — that is the part that is actually new.

Put in order

Put the moves of Derive MAP: drop the constant, take the log into the order they have to happen.

1. Start from the posterior
2. Drop p(D) — it does not depend on θ
3. Take logs — log is monotonic

Why: These are the moves of the worked example in the order it makes them, and each one is set up by the one before it. MAP is the arg max of the posterior over θ — the most probable parameter given the data.

Annotate

From Derive MAP: drop the constant, take the log — read this one piece at a time. What is each part doing?

On: \( \theta_{\text{MAP}} = \arg\max_{\theta}\; \frac{p(D\mid\theta)\,p(\theta)}{p(D)} \)

• MAP is the arg max of the posterior over θ — the most probable parameter given the data.
• The denominator p(D) is a fixed number once the data is in; scaling by a positive constant never moves an arg max. So the posterior is PROPORTIONAL to likelihood × prior.
• log is strictly increasing, so arg max is unchanged; and log turns the product into a SUM, which is far easier to differentiate.

Discussion prompt

Explain The one-line summary: MAP = MLE + log-prior to a student a year behind you. No notation, no jargon they have not met — and it still has to be true.

Hint: If your explanation needs a symbol they have never seen, you are describing the notation rather than the idea.

Discussion prompt

A tiny MAP by hand — one weight — before any calculation: what is the plan? Name the moves in order, in plain English, without doing the arithmetic.

Hint: It starts with: Sums: Σxᵢ² = 1+4+9 = 14, Σxᵢyᵢ = 2.1+7.8+18.6 = 28.5

Comparison matrix

From A tiny MAP by hand — one weight: every row here is a choice with a cost. Fill the denominator column, then say which row you would actually pick and what you give up for it.

Predict first

Confirm the by-hand numbers with a runnable snippet. The MAP weight is the same two sums with σ²/τ² added to the denominator:

Commit before you compute: what does The tiny MAP in code come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: λ = σ²/τ² = 4 pulls the estimate from 2.0357 toward 0

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. Adding λ to the denominator is exactly the 1-D case of (XᵀX + λI).

Step through it

Step through The tiny MAP in code one row at a time. What is driving the change, and what would the row after the last one be?

1. Step 1: print is Sxx, Sxy
2. Step 2: print is MLE
3. Step 3: print is MAP

Discussion prompt

The Gaussian log-prior is a negative quadratic — before any calculation: what is the plan? Name the moves in order, in plain English, without doing the arithmetic.

Hint: It starts with: Density: p(w) ∝ exp(−‖w‖² / (2τ²))

Discussion prompt

Work backwards. The example finished here:

Numeric check (τ²=0.5): the quadratic part is w²/(2τ²)

What was it asked to do, and what must it have been given? Reconstruct the problem from its answer.

Hint: Every quantity in the result had to enter somewhere. Account for each one.

Put in order

Put the moves of Add the two −logs → the ridge objective into the order they have to happen.

1. MAP minimizes −(log-likelihood + log-prior)
2. Multiply through by 2σ² (a positive constant)
3. Name the ratio λ = σ²/τ² — this IS ridge

Why: These are the moves of the worked example in the order it makes them, and each one is set up by the one before it. Flip the arg max of the sum of logs into an arg min of the negative sum.

Draw it

Translate both ways. First write the expression above as a sentence with no symbols in it at all. Then cover it, and write your sentence back as notation. If the two versions disagree, the disagreement is the thing to fix.

Discussion prompt

Ridge's normal equations are (XᵀX + λI)w = Xᵀy (Lesson 7). Solve them with λ = 1 on the student data and compare to plain OLS. Runnable as written:

What do you need to know — or decide — before the first line can be written? List everything the problem has to hand you.

Hint: Anything you would have to invent to get started is a thing the problem must supply.

Predict first

A student writes this, and it looks reasonable:

A tighter prior (small τ²) should mean less penalty, so put the prior variance on top: λ = τ²/σ².

It is wrong. Say what breaks — and say it before you turn the page.

Correct: This gets it backwards. A TIGHTER prior (small τ²) is a STRONGER belief that weights are small, so it must penalize MORE — λ should grow as τ² shrinks.

Read it straight off the derivation: after clearing the loss fraction, the penalty coefficient is the noise variance over the prior variance.

Why: This gets it backwards. A TIGHTER prior (small τ²) is a STRONGER belief that weights are small, so it must penalize MORE — λ should grow as τ² shrinks. τ²/σ² shrinks as τ²→0, giving less penalty exactly when you wanted more.

Discussion prompt

Read the picture before the words. What is this showing, and what is the one thing it is built to make obvious? Commit to an answer, then read on.

Hint: Name the parts, then say what changes between them — and if nothing changes, say what is being held still.

Discussion prompt

The Laplace log-prior is proportional to |w| — before any calculation: what is the plan? Name the moves in order, in plain English, without doing the arithmetic.

Hint: It starts with: −log p(wⱼ) = |wⱼ|/b + log(2b)

Step through it

Step through The Laplace log-prior is proportional to |w| one row at a time. One of these columns never changes — find it, and say why it cannot.

1. Step 1: w is −2
2. Step 2: w is −1
3. Step 3: w is 0
4. Step 4: w is 1
5. Step 5: w is 2

Sort into buckets

These are the pieces of Lesson 12: MAP, Priors & Regularization, out of order. Put each one back under the part of the lesson it belongs to.

MAP: a prior is a belief

The running example: fit a line, but small data; MLE recap: maximize the likelihood; A prior is what you believe before the data

Gaussian prior → L2 / ridge

The Gaussian noise likelihood (from Lesson 8); The Gaussian log-prior is a negative quadratic; Add the two −logs → the ridge objective

Laplace prior → L1 / lasso

The Laplace prior spikes at zero; The Laplace log-prior is proportional to |w|; Spike at zero, fat tails — the best of both

s1

MAP: a prior is a belief is where Lesson 12: MAP, Priors & Regularization puts The running example: fit a line, but small data, MLE recap: maximize the likelihood, A prior is what you believe before the data. Knowing which part of the lesson a problem belongs to is most of knowing which method to reach for.

s2

Gaussian prior → L2 / ridge is where Lesson 12: MAP, Priors & Regularization puts The Gaussian noise likelihood (from Lesson 8), The Gaussian log-prior is a negative quadratic, Add the two −logs → the ridge objective. Knowing which part of the lesson a problem belongs to is most of knowing which method to reach for.

s3

Laplace prior → L1 / lasso is where Lesson 12: MAP, Priors & Regularization puts The Laplace prior spikes at zero, The Laplace log-prior is proportional to |w|, Spike at zero, fat tails — the best of both. Knowing which part of the lesson a problem belongs to is most of knowing which method to reach for.

Predict first

The table runs: 1.0 | 2.0000 | 1.0000 · 0.1 | 0.2000 | 1.0000 · 0.01 | 0.0200 | 1.0000

In Watch the two gradients near zero, given the rows so far: what is the next one — the row where w is 0.001?

Correct: 0.001 | 0.0020 | 1.0000

Why: The relationship between the columns, not the individual numbers, is what generates the next row. The L2 restoring force vanishes near the origin (proportional shrink), so the coefficient asymptotes to a tiny nonzero value.

Fill in the blanks

From One coordinate, exactly: shrink vs soft-threshold — one line has had its right-hand side removed. Put it back.

import numpy as np
def soft(rho, t):
return np.sign(rho)*max(abs(rho)-t, 0.0)
lam = 0.4
for rho in [1.0, 0.5, 0.15, 0.10, 0.05]:
ridge = rho/(1+lam)
lasso = soft(rho, lam/2) # threshold lam/2 = 0.2
print(f'rho=___ ridge=___ lasso=___')

Why: ridge is what everything below it consumes, so the wrong expression here fails later and somewhere else. Soft-thresholding subtracts λ/2 and clips at 0, so any small coordinate is killed outright.

Sort into buckets

Sort these by lasso = soft(ρ, 0.2), from memory, without looking back at One coordinate, exactly: shrink vs soft-threshold. Telling them apart on the spot is the skill; the table is only where the answer happens to be written down.

0.8000

1.00

0.3000

0.50

0.0000 ← zero

0.15; 0.10; 0.05

g1

lasso = soft(ρ, 0.2) is "0.8000" for 1.00 — that is what the table on "One coordinate, exactly: shrink vs…" records, and it is the single property separating this group from the rest.

g2

lasso = soft(ρ, 0.2) is "0.3000" for 0.50 — that is what the table on "One coordinate, exactly: shrink vs…" records, and it is the single property separating this group from the rest.

g3

lasso = soft(ρ, 0.2) is "0.0000 ← zero" for 0.15, 0.10, 0.05 — that is what the table on "One coordinate, exactly: shrink vs…" records, and it is the single property separating this group from the rest.

Predict first

A student writes this, and it looks reasonable:

Ridge penalizes weight size, so it should zero out the useless features — just like lasso, only smoother.

It is wrong. Say what breaks — and say it before you turn the page.

Correct: L2's gradient 2λw shrinks PROPORTIONALLY and vanishes as w→0, so it never reaches zero.

Ridge shrinks every coefficient smoothly; only L1 / lasso sets coordinates to exactly zero.

Why: L2's gradient 2λw shrinks PROPORTIONALLY and vanishes as w→0, so it never reaches zero. On our 10-feature demo ridge leaves ALL 10 coefficients nonzero (the smallest is still about ±0.003).

Discussion prompt

The rule this trap just fixed:

Verified in Part 5: on 10 features (only 0,3,7 real) ridge keeps 10 nonzero, lasso zeros 7 and recovers exactly the true support {0,3,7}.

Now break it on purpose. Build a case that violates it and follow the consequences until something visibly fails. Where does the failure first show up — and would you have noticed it if you had not been looking?

Hint: The dangerous rules are the ones whose violation still produces an answer. If yours fails loudly, try to find one that fails quietly.

Put in order

Put the moves of Beta × Binomial = Beta, by matching exponents into the order they have to happen.

1. Write the two kernels (drop constants)
2. Add the exponents
3. Recognize the Beta(α+k, β+n−k) kernel

Why: These are the moves of the worked example in the order it makes them, and each one is set up by the one before it. Binomial likelihood in the success prob p is pᵏ(1−p)ⁿ⁻ᵏ; Beta(α,β) prior kernel is p^(α−1)(1−p)^(β−1).

Annotate

From Beta × Binomial = Beta, by matching exponents — read this one piece at a time. What is each part doing?

On: \( p(p\mid D) = \mathrm{Beta}(\alpha + k,\; \beta + n - k) \)

• Binomial likelihood in the success prob p is pᵏ(1−p)ⁿ⁻ᵏ; Beta(α,β) prior kernel is p^(α−1)(1−p)^(β−1).
• Same base p multiplies by adding exponents: k+(α−1) and (n−k)+(β−1).
• That is exactly the kernel of a Beta with parameters α+k and β+n−k. The posterior is Beta again — conjugacy proved: add successes to α, failures to β.

Predict first

Prior Beta(2,2), data k=7 successes in n=10. The claim: posterior is Beta(9,5), i.e. likelihood × prior is proportional to the Beta(9,5) kernel with the SAME ratio at every p. Check it:

Commit before you compute: what does Verify conjugacy numerically come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: The ratio is a CONSTANT (1.0) across all p → same distribution

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. likelihood×prior and the Beta(9,5) kernel differ only by a normalizing constant, so their ratio is identical everywhere.

Step through it

Step through Verify conjugacy numerically and watch the numbers move. Now imagine the input ten times bigger: which column is the one that stops this being practical?

1. Step 1: p is 0.3
2. Step 2: p is 0.5
3. Step 3: p is 0.7
4. Step 4: p is 0.9

Fill in the blanks

From Dirichlet–Categorical: the same trick, K outcomes — one line has had its right-hand side removed. Put it back.

import numpy as np
alpha0 = np.array([1., 1., 1.]) # flat Dirichlet prior
counts = np.array([12., 5., 3.]) # 20 rolls
post = alpha0 + counts # Dirichlet(13, 6, 4)
print('posterior alpha:', post)
print('posterior mean :', (post/post.sum()).round(4))
print('MLE :', (counts/counts.sum()).round(4))

Why: counts is what everything below it consumes, so the wrong expression here fails later and somewhere else. Just as Beta added successes/failures, Dirichlet adds each outcome's count to its α.

Step through it

Step through Dirichlet–Categorical: the same trick, K outcomes one row at a time. What is driving the change, and what would the row after the last one be?

1. Step 1: face is 1
2. Step 2: face is 2
3. Step 3: face is 3

Discussion prompt

Explain Dropout: randomly delete neurons to a student a year behind you. No notation, no jargon they have not met — and it still has to be true.

Hint: If your explanation needs a symbol they have never seen, you are describing the notation rather than the idea.

Comparison matrix

From How big is that ensemble?: refill the distinct sub-networks 2ⁿ column from what you know. The rest of the table is as it appeared.

Fill in the blanks

From Sparse-truth data: only 3 of 10 features matter — one line has had its right-hand side removed. Put it back.

import numpy as np
rng = np.random.default_rng(1)
X = rng.normal(size=(50, 10))
true = np.zeros(10); true[[0, 3, 7]] = [3., -2., 1.5]
y = X @ true + 0.1*rng.normal(size=50)
print('true nonzero idx:', np.flatnonzero(true))
print('X shape:', X.shape)

Why: X is what everything below it consumes, so the wrong expression here fails later and somewhere else. The true weight vector is zero everywhere except three slots.

Discussion prompt

Fit Ridge(alpha=1.0) and Lasso(alpha=0.1) on the same data and count how many coefficients each drives to exactly zero:

What do you need to know — or decide — before the first line can be written? List everything the problem has to hand you.

Hint: Anything you would have to invent to get started is a thing the problem must supply.

Discussion prompt

Work backwards. The example finished here:

ridge: 0 exact zeros; lasso: 7 exact zeros, keeps exactly {0, 3, 7}

What was it asked to do, and what must it have been given? Reconstruct the problem from its answer.

Hint: Every quantity in the result had to enter somewhere. Account for each one.

Predict first

λ (sklearn's alpha) is the prior strength — crank it up and more coordinates snap to zero. Sweep it and count the survivors:

Commit before you compute: what does The lasso path: λ is the sparsity dial come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: Too small (0.01) keeps noise features; the sweet spot 0.1–1.0 recovers exactly {0,3,7}; too big (3.0) kills everything

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. A weak prior barely thresholds, so noise coordinates survive; a strong prior over-shrinks and zeros even the true features.

Sort into buckets

Sort these by # nonzero, from memory, without looking back at The lasso path: λ is the sparsity dial. Telling them apart on the spot is the skill; the table is only where the answer happens to be written down.

5

0.01

3

0.1; 0.5; 1.0

0

3.0

g1

# nonzero is "5" for 0.01 — that is what the table on "The lasso path: λ is the sparsity dial" records, and it is the single property separating this group from the rest.

g2

# nonzero is "3" for 0.1, 0.5, 1.0 — that is what the table on "The lasso path: λ is the sparsity dial" records, and it is the single property separating this group from the rest.

g3

# nonzero is "0" for 3.0 — that is what the table on "The lasso path: λ is the sparsity dial" records, and it is the single property separating this group from the rest.

Predict first

A student writes this, and it looks reasonable:

They are both just "regularization", so pick whichever — they do the same job, and I need to drop useless features, so ridge is fine.

It is wrong. Say what breaks — and say it before you turn the page.

Correct: Ridge never eliminates a feature — on our data it returns all 10 nonzero coefficients.

Match the penalty to the goal: L2 for stability/shrinkage, L1 for sparsity/feature selection.

Why: Ridge never eliminates a feature — on our data it returns all 10 nonzero coefficients. If the goal is a sparse, interpretable model, L2 is the wrong tool: you get a dense weight vector, not a shortlist.

Sort into buckets

Some of these hold up and some are the exact mistakes this lesson is built to prevent. Sort them.

Holds up

In Lesson 8 the maximum-likelihood estimate picked the θ that makes the observed data most probable. For Gaussian noise on a linear model, that is exactly least squares.; Bayes' rule flips a conditional. Applied to a parameter θ given data D, the posterior is the likelihood times the prior, divided by a normalizing constant:; The boxed result is the spine of the whole lesson. MAP is MLE with one extra additive term — the log of the prior:

Breaks

A tighter prior (small τ²) should mean less penalty, so put the prior variance on top: λ = τ²/σ².; Ridge penalizes weight size, so it should zero out the useless features — just like lasso, only smoother.

sound

These are stated as this lesson states them — each one survives the edge cases Lesson 12: MAP, Priors & Regularization puts it through.

flawed

Each of these is lifted from a trap in this deck: reasonable-sounding, and wrong in a way that only shows up once you rely on it.

Predict first

The table runs: sklearn Ridge | 2.844950 | −0.030003 | 0.012108 · closed form | 2.844950 | −0.030003 | 0.012108

In Verify: the closed form is really MAP, given the rows so far: what is the next one — the row where source is allclose?

Correct: allclose | True | — | —

Why: The relationship between the columns, not the individual numbers, is what generates the next row. sklearn's alpha is our λ. Solving (XᵀX+λI)w = Xᵀy reproduces it exactly, closing the loop: the Gaussian log-prior and the L2 penalty are the same math.

Discussion prompt

Explain Verify: the closed form is really MAP by analogy to something with no Machine Learning in it at all — a queue, a recipe, a map, a bank balance, whatever fits. Then say where your analogy breaks.

Hint: An analogy that never breaks is not an analogy, it is the same idea wearing a hat. Find the seam — that is the part that is actually new.

Step through it

Step through When does regularization actually help? one row at a time. What is driving the change, and what would the row after the last one be?

1. Step 1: model is OLS
2. Step 2: model is ridge (α=1)
3. Step 3: model is lasso (α=0.1)

Put in order

These are the steps of Regularization-as-prior recipe, scrambled. Put them back in order before the next slide shows you.

1. Likelihood → loss: write −log p(data|θ) (MSE for Gaussian noise, cross-entropy for classification)
2. Prior → penalty: add −log p(θ) — the belief, in log form
3. Gaussian prior ⇒ λ‖w‖² (ridge / weight decay), with λ = σ²/τ² — shrinks, never zeros
4. Laplace prior ⇒ λ‖w‖₁ (lasso) — constant pull ⇒ soft-threshold ⇒ exact zeros
5. Conjugate prior (Beta, Dirichlet) ⇒ posterior in the same family, update by adding counts
6. Tune λ = prior strength: large λ = confident prior = more shrinkage; λ → 0 = back to MLE

Why: This is the order the recipe itself gives. Recalling the sequence without the slide in front of you is the difference between recognising the method and being able to run it — most of what goes wrong in practice is a step done out of turn.

Discussion prompt

Regularization-as-prior recipe works on the cases you have just seen. Push it to the edge: what is the most degenerate input it still handles — empty, zero, one item, everything equal — and what is the first case where it stops being true? Name the case, not just "it breaks".

Hint: Try the smallest legal input, then the largest, then the one where two things collide. Methods are specified at their edges; the middle takes care of itself.

Eliminate the wrong options

L2 regularization (ridge) is the MAP estimate under which prior on the weights?

3 of these 4 are wrong. Strike them one at a time, and say what rules each one out before you strike the next. The survivor is the answer.

• A. A zero-mean Gaussian prior
• B. A Laplace prior
• C. A uniform (flat) prior
• D. A Beta prior

Survives elimination: A

Why: −log of a zero-mean Gaussian is proportional to ‖w‖², so a Gaussian prior adds exactly the L2 penalty. Minimizing loss + λ‖w‖² is ridge, and its strength is λ = σ²/τ².

Check your understanding

L2 regularization (ridge) is the MAP estimate under which prior on the weights?

• A. A zero-mean Gaussian prior (correct)
• B. A Laplace prior
• C. A uniform (flat) prior
• D. A Beta prior

Answer: A

Why: −log of a zero-mean Gaussian is proportional to ‖w‖², so a Gaussian prior adds exactly the L2 penalty. Minimizing loss + λ‖w‖² is ridge, and its strength is λ = σ²/τ².

Why B tempts people

A Laplace prior gives the L1 penalty (lasso), whose −log is proportional to |w| — the sparse one, not ridge.

Why C tempts people

A uniform prior adds a constant (its log doesn't depend on w), so it contributes NO penalty — MAP collapses back to plain MLE / OLS.

Why D tempts people

A Beta prior lives on a probability in [0,1] (Bernoulli parameter), not on real-valued regression weights that can be negative or large.

Predict first

Why does L1 (lasso) set coefficients to EXACTLY zero while L2 (ridge) does not?

Answer it in your own words, now, with nothing to choose from. The options are on the next slide — and picking the right one off a list is an easier skill than producing it.

Correct: The L1 penalty's gradient magnitude stays constant (λ) as w → 0, so it pushes a small weight all the way to 0; L2's gradient 2λw fades to 0 and never finishes the job

Why: The subgradient of λ|w| has constant magnitude λ (its sign), so the pull toward 0 never weakens — the soft-threshold snaps small coordinates to exactly 0. L2's gradient 2λw shrinks with w and vanishes at the origin, so ridge only asymptotes toward 0.

Check your understanding

Why does L1 (lasso) set coefficients to EXACTLY zero while L2 (ridge) does not?

• A. The L1 penalty's gradient magnitude stays constant (λ) as w → 0, so it pushes a small weight all the way to 0; L2's gradient 2λw fades to 0 and never finishes the job (correct)
• B. L1 uses a larger λ than L2 by default
• C. L1 is non-convex, so it has multiple minima including zero
• D. L2 rounds tiny coefficients up, keeping them nonzero

Answer: A

Why: The subgradient of λ|w| has constant magnitude λ (its sign), so the pull toward 0 never weakens — the soft-threshold snaps small coordinates to exactly 0. L2's gradient 2λw shrinks with w and vanishes at the origin, so ridge only asymptotes toward 0.

Why B tempts people

λ is a free hyperparameter for both; sparsity is about the SHAPE of the penalty near 0 (constant vs proportional gradient), not the size of λ.

Why C tempts people

The L1 penalty is convex (it's |w|), just non-differentiable at 0. Sparsity comes from that kink, not from non-convexity or multiple minima.

Why D tempts people

There is no rounding step. L2 keeps coefficients nonzero because its restoring force fades near 0, not because of any rounding.

Predict first

With a Beta(α, β) prior and k successes in n Bernoulli trials, the posterior is…

Answer it in your own words, now, with nothing to choose from. The options are on the next slide — and picking the right one off a list is an easier skill than producing it.

Correct: Beta(α + k, β + n − k)

Why: Multiplying pᵏ(1−p)ⁿ⁻ᵏ by the Beta kernel p^(α−1)(1−p)^(β−1) adds exponents: α gets the successes k, β gets the failures n−k. The posterior is Beta(α+k, β+n−k) — that's conjugacy.

Check your understanding

With a Beta(α, β) prior and k successes in n Bernoulli trials, the posterior is…

• A. Beta(α + k, β + n − k) (correct)
• B. Beta(α + n, β + k)
• C. Gaussian(k/n, 1)
• D. Dirichlet(α + k)

Answer: A

Why: Multiplying pᵏ(1−p)ⁿ⁻ᵏ by the Beta kernel p^(α−1)(1−p)^(β−1) adds exponents: α gets the successes k, β gets the failures n−k. The posterior is Beta(α+k, β+n−k) — that's conjugacy.

Why B tempts people

Swaps the updates — α must receive the successes k and β the failures (n−k), not n and k. Beta(α+n, β+k) mislabels which count goes where.

Why C tempts people

The Beta–Bernoulli posterior stays Beta; conjugacy PRESERVES the family, it doesn't switch to a Gaussian.

Why D tempts people

Dirichlet is the conjugate prior for the CATEGORICAL/multinomial likelihood (many outcomes), not for a single Bernoulli probability.

Eliminate the wrong options

In the ridge MAP objective ‖Xw−y‖² + λ‖w‖² with λ = σ²/τ², what happens as the prior variance τ² → ∞?

3 of these 4 are wrong. Strike them one at a time, and say what rules each one out before you strike the next. The survivor is the answer.

• A. λ → 0, so MAP reduces to plain MLE / OLS
• B. λ → ∞, so all weights are forced to 0
• C. The prior becomes Laplace, giving lasso
• D. The posterior becomes exactly the prior

Survives elimination: A

Why: λ = σ²/τ² → 0 as τ² → ∞. A prior with infinite variance is flat (uninformative), so its log-term is constant and MAP becomes MLE — plain least squares, w = [2.2, 0.6] on our data.

Check your understanding

In the ridge MAP objective ‖Xw−y‖² + λ‖w‖² with λ = σ²/τ², what happens as the prior variance τ² → ∞?

• A. λ → 0, so MAP reduces to plain MLE / OLS (correct)
• B. λ → ∞, so all weights are forced to 0
• C. The prior becomes Laplace, giving lasso
• D. The posterior becomes exactly the prior

Answer: A

Why: λ = σ²/τ² → 0 as τ² → ∞. A prior with infinite variance is flat (uninformative), so its log-term is constant and MAP becomes MLE — plain least squares, w = [2.2, 0.6] on our data.

Why B tempts people

That is the OPPOSITE limit: λ → ∞ happens as τ² → 0 (a tight prior at 0), which crushes the weights. A wide prior barely penalizes.

Why C tempts people

Changing τ² only rescales the Gaussian; it never turns it into a Laplace. The penalty stays L2 for any Gaussian prior.

Why D tempts people

With τ² → ∞ the data dominates, so the posterior concentrates at the MLE — the prior's influence VANISHES rather than taking over.

Discussion prompt

Build rules: type every line, set the RNG seed default_rng(1) so results are reproducible, run after each block, and compare coefficients to the true support [0, 3, 7].

That is stated as though it always holds. Do one of two things: produce a case where it fails, or say precisely what rules such a case out. "It just does" is not on the menu.

Hint: Hunt at the extremes first — zero, one, negative, empty, equal. If every extreme survives, the reason they survive is the proof.

Comparison matrix

From Milestone 3 — count the zeros: every row here is a choice with a cost. Fill the exact zeros (verified) column, then say which row you would actually pick and what you give up for it.

Comparison matrix

From The full program: refill the value (verified) column from what you know. The rest of the table is as it appeared.

Draw it

One page, no notation unless you need it: draw how these connect — MAP: a prior is a belief · Gaussian prior → L2 / ridge · Laplace prior → L1 / lasso · Conjugate priors · Dropout as approximate Bayes · Ridge vs Lasso — the real fit. Put an arrow wherever one of them is what makes another possible, and label the arrow with why.