Lesson 53: Gaussian Mixture Models & the EM Algorithm

Discussion prompt

Before we open Lesson 53: Gaussian Mixture Models & the EM Algorithm: without looking back, what was the main idea of k-Means Clustering, and what could you do by the end of it that you could not do before?

Hint: One sentence for the idea, one for the skill. If the second one is blank, that is the part to revisit.

Discussion prompt

A GMM is a weighted sum of K Gaussian components. Each component k has weight pi_k, mean mu_k, and covariance Sigma_k.

That is stated as though it always holds. Do one of two things: produce a case where it fails, or say precisely what rules such a case out. "It just does" is not on the menu.

Hint: Hunt at the extremes first — zero, one, negative, empty, equal. If every extreme survives, the reason they survive is the proof.

Comparison matrix

From Covariance types — four flavors: refill the shape column from what you know. The rest of the table is as it appeared.

Comparison matrix

From Why soft assignments beat hard ones: every row here is a choice with a cost. Fill the k-Means hard column, then say which row you would actually pick and what you give up for it.

Discussion prompt

Explain E-step: posterior responsibility by analogy to something with no Machine Learning in it at all — a queue, a recipe, a map, a bank balance, whatever fits. Then say where your analogy breaks.

Hint: An analogy that never breaks is not an analogy, it is the same idea wearing a hat. Find the seam — that is the part that is actually new.

Put in order

Put the moves of E-step by hand (K=2, 1-D) into the order they have to happen.

1. Compute pi_1 * N(0.5; -2.0, 0.7) = 0.35 * 0.00115 = 0.000402
2. Compute pi_2 * N(0.5; +2.0, 0.7) = 0.65 * 0.10917 = 0.070960
3. p(x=0.5) = 0.000402 + 0.070960 = 0.071362; r1 = 0.0056, r2 = 0.9944

Why: These are the moves of the worked example in the order it makes them, and each one is set up by the one before it. The left component's PDF at x=0.5 is tiny: 2.5 standard deviations away.

Step through it

Step through E-step by hand (K=2, 1-D) one row at a time. What is driving the change, and what would the row after the last one be?

1. Step 1: x is -2.0
2. Step 2: x is +0.0
3. Step 3: x is +2.0

Discussion prompt

Explain M-step: weighted MLE to a student a year behind you. No notation, no jargon they have not met — and it still has to be true.

Hint: If your explanation needs a symbol they have never seen, you are describing the notation rather than the idea.

Put in order

Put the moves of M-step by hand (N=4 toy) into the order they have to happen.

1. N_k1 = 0.92 + 0.88 + 0.05 + 0.03 = 1.88; N_k2 = 4 - 1.88 = 2.12
2. mu_k1_new = (0.92-2.5 + 0.88-1.8 + 0.051.5 + 0.032.3) / 1.88 = -1.9894
3. pi_k1_new = 1.88 / 4 = 0.4700; pi_k2_new = 2.12 / 4 = 0.5300

Why: These are the moves of the worked example in the order it makes them, and each one is set up by the one before it. Effective count for each cluster — not an integer because assignments are soft.

Discussion prompt

Work backwards. The example finished here:

pi_k1_new = 1.88 / 4 = 0.4700; pi_k2_new = 2.12 / 4 = 0.5300

What was it asked to do, and what must it have been given? Reconstruct the problem from its answer.

Hint: Every quantity in the result had to enter somewhere. Account for each one.

Predict first

A student writes this, and it looks reasonable:

The GMM log-likelihood is non-convex, but EM is guaranteed to converge — so it finds the global maximum.

It is wrong. Say what breaks — and say it before you turn the page.

Correct: EM guarantees the log-likelihood never decreases, so it converges — but only to a local maximum.

EM guarantees monotone non-decreasing log-likelihood, not the global maximum.

Why: EM guarantees the log-likelihood never decreases, so it converges — but only to a local maximum. The non-convex landscape has many local optima, and a bad initialization can land in a poor one.

Discussion prompt

The rule this trap just fixed:

EM guarantees monotone non-decreasing log-likelihood, not the global maximum.

Now break it on purpose. Build a case that violates it and follow the consequences until something visibly fails. Where does the failure first show up — and would you have noticed it if you had not been looking?

Hint: The dangerous rules are the ones whose violation still produces an answer. If yours fails loudly, try to find one that fails quietly.

Predict first

Run GaussianMixture with max_iter capped at increasing values and read off the log-likelihood after each cap.

Commit before you compute: what does EM convergence trace (sklearn) come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: Iter 1: -1120.32, iter 2: -1119.92, iter 3: -1119.77, iter 5: -1119.67, iter 10: -1119.67 (converged)

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. The log-likelihood is strictly non-decreasing and flattens after ~5 iterations on this clean data.

Sort into buckets

These are the pieces of Lesson 53: Gaussian Mixture Models & the EM Algorithm, out of order. Put each one back under the part of the lesson it belongs to.

The GMM density model

A mixture of K Gaussians; Covariance types — four flavors; Why soft assignments beat hard ones

E-step — soft responsibilities

E-step: posterior responsibility; E-step by hand (K=2, 1-D)

M-step — update parameters

M-step: weighted MLE; M-step by hand (N=4 toy); EM convergence trace (sklearn)

s1

The GMM density model is where Lesson 53: Gaussian Mixture Models & the EM Algorithm puts A mixture of K Gaussians, Covariance types — four flavors, Why soft assignments beat hard ones. Knowing which part of the lesson a problem belongs to is most of knowing which method to reach for.

s2

E-step — soft responsibilities is where Lesson 53: Gaussian Mixture Models & the EM Algorithm puts E-step: posterior responsibility, E-step by hand (K=2, 1-D). Knowing which part of the lesson a problem belongs to is most of knowing which method to reach for.

s3

M-step — update parameters is where Lesson 53: Gaussian Mixture Models & the EM Algorithm puts M-step: weighted MLE, M-step by hand (N=4 toy), EM convergence trace (sklearn). Knowing which part of the lesson a problem belongs to is most of knowing which method to reach for.

Discussion prompt

Explain From-scratch EM: the 20-line loop to a student a year behind you. No notation, no jargon they have not met — and it still has to be true.

Hint: If your explanation needs a symbol they have never seen, you are describing the notation rather than the idea.

Predict first

K=2, make_blobs(200, centers=2, std=0.8, seed=0). Initialize means at two random data points; covs = I; pis = [0.5, 0.5]. Track LL across iterations.

Commit before you compute: what does EM from scratch — iteration trace come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: After 20 iterations: means = [[0.964, 4.356], [1.954, 0.762]] — matches sklearn (differs by <0.01)

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. The from-scratch loop converges to the same local optimum as sklearn's optimized C++ backend, confirming the math is correct.

Comparison matrix

From GMM vs k-Means: when does GMM win?: refill the assignment type column from what you know. The rest of the table is as it appeared.

Predict first

A student writes this, and it looks reasonable:

diagonal covariance is cheaper than full and still models different variances per feature — it should work fine on any blob-like clusters.

It is wrong. Say what breaks — and say it before you turn the page.

Correct: Diagonal covariance forces the fitted ellipses to be axis-aligned — it cannot represent off-diagonal correlation.

Use covariance_type='full' when the data may have correlations between features.

Why: Diagonal covariance forces the fitted ellipses to be axis-aligned — it cannot represent off-diagonal correlation. On data generated by a full covariance matrix, it makes the same error as spherical, just with different axis scales.

Sort into buckets

Some of these hold up and some are the exact mistakes this lesson is built to prevent. Sort them.

Holds up

A GMM is a weighted sum of K Gaussian components. Each component k has weight pi_k, mean mu_k, and covariance Sigma_k.; K-Means (Lesson 21) commits: every point is 100% in one cluster. A point near the boundary gets assigned exactly as confidently as a point at the center.; Given current parameters, the responsibility r_{ik} is the posterior probability that component k generated point i — just Bayes' rule.

Breaks

The GMM log-likelihood is non-convex, but EM is guaranteed to converge — so it finds the global maximum.; diagonal covariance is cheaper than full and still models different variances per feature — it should work fine on any blob-like clusters.

sound

These are stated as this lesson states them — each one survives the edge cases Lesson 53: Gaussian Mixture Models & the EM Algorithm puts it through.

flawed

Each of these is lifted from a trap in this deck: reasonable-sounding, and wrong in a way that only shows up once you rely on it.

Predict first

Fit GMM with K = 1 through 6 (full covariance) on make_blobs(300, centers=3, std=0.8, seed=42). The true K=3 should win on BIC.

Commit before you compute: what does BIC for selecting K and covariance type come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: K=1: 3753.0, K=2: 2600.0, K=3: 2150.3 (minimum), K=4: 2171.7, K=5: 2202.3

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. BIC drops steeply from K=1 to K=3 (large fit gain) then rises (penalty exceeds marginal fit gain).

Discussion prompt

Work backwards. The example finished here:

K=1: 3753.0, K=2: 2600.0, K=3: 2150.3 (minimum), K=4: 2171.7, K=5: 2202.3

What was it asked to do, and what must it have been given? Reconstruct the problem from its answer.

Hint: Every quantity in the result had to enter somewhere. Account for each one.

Discussion prompt

Explain Anomaly detection with score_samples by analogy to something with no Machine Learning in it at all — a queue, a recipe, a map, a bank balance, whatever fits. Then say where your analogy breaks.

Hint: An analogy that never breaks is not an analogy, it is the same idea wearing a hat. Find the seam — that is the part that is actually new.

Put in order

These are the steps of The GMM / EM recipe, scrambled. Put them back in order before the next slide shows you.

1. Model: p(x) = sum_k pi_k * N(x; mu_k, Sigma_k), one row in the GMM density
2. E-step: r_{ik} = pi_k N(x_i; mu_k, Sig_k) / p(x_i) — Bayes posterior over components
3. M-step: weighted MLE — mu_k = sum_i(r_{ik} x_i) / N_k, Sigma_k similarly
4. Convergence: LL never decreases; run multiple restarts (n_init >= 5) for robustness
5. Model selection: BIC over K in {1..10} and covariance type; pick the minimum
6. Anomaly detection: fit GMM on clean data; flag score_samples(X) < threshold

Why: This is the order the recipe itself gives. Recalling the sequence without the slide in front of you is the difference between recognising the method and being able to run it — most of what goes wrong in practice is a step done out of turn.

Discussion prompt

The GMM / EM recipe works on the cases you have just seen. Push it to the edge: what is the most degenerate input it still handles — empty, zero, one item, everything equal — and what is the first case where it stops being true? Name the case, not just "it breaks".

Hint: Try the smallest legal input, then the largest, then the one where two things collide. Methods are specified at their edges; the middle takes care of itself.

Eliminate the wrong options

In the E-step, r_{ik} is the responsibility of component k for point i. What quantity does the denominator sum_j pi_j N(x_i; mu_j, Sigma_j) equal?

3 of these 4 are wrong. Strike them one at a time, and say what rules each one out before you strike the next. The survivor is the answer.

• A. The marginal density p(x_i) under the GMM
• B. The joint probability p(x_i, k)
• C. The prior pi_k for component k
• D. The log-likelihood of the full dataset

Survives elimination: A

Why: Summing pi_j * N(x_i; mu_j, Sigma_j) over all j marginalizes out the latent component assignment — this is exactly the GMM marginal density p(x_i). Dividing by p(x_i) turns the joint pi_k*N into a posterior (Bayes' theorem).

Check your understanding

In the E-step, r_{ik} is the responsibility of component k for point i. What quantity does the denominator sum_j pi_j N(x_i; mu_j, Sigma_j) equal?

• A. The marginal density p(x_i) under the GMM (correct)
• B. The joint probability p(x_i, k)
• C. The prior pi_k for component k
• D. The log-likelihood of the full dataset

Answer: A

Why: Summing pi_j * N(x_i; mu_j, Sigma_j) over all j marginalizes out the latent component assignment — this is exactly the GMM marginal density p(x_i). Dividing by p(x_i) turns the joint pi_k*N into a posterior (Bayes' theorem).

Why B tempts people

The joint p(x_i, k) = pi_k * N(x_i; mu_k, Sigma_k) — that's the numerator, not the denominator.

Why C tempts people

pi_k is the prior for component k and appears only in the numerator as the weight for component k specifically.

Why D tempts people

The dataset log-likelihood is a sum over all N points; the denominator here is just the marginal for one point x_i.

Predict first

After the E-step on N=200 points with K=3, the soft counts are N_1=72, N_2=83, N_3=45. What is the new mixing weight pi_3?

Answer it in your own words, now, with nothing to choose from. The options are on the next slide — and picking the right one off a list is an easier skill than producing it.

Correct: 0.225

Why: pi_k_new = N_k / N = 45 / 200 = 0.225. The M-step weight update is just the fraction of total soft count that component k holds.

Check your understanding

After the E-step on N=200 points with K=3, the soft counts are N_1=72, N_2=83, N_3=45. What is the new mixing weight pi_3?

• A. 0.225 (correct)
• B. 0.415
• C. 0.333
• D. 0.045

Answer: A

Why: pi_k_new = N_k / N = 45 / 200 = 0.225. The M-step weight update is just the fraction of total soft count that component k holds.

Why B tempts people

0.415 = N_2/N = 83/200 — that is the update for component 2, not component 3.

Why C tempts people

0.333 would be correct if all components had equal soft count (N=200/3). That's only true at a fully symmetric solution.

Why D tempts people

0.045 would equal N_3 divided by N*10 — a factor-of-10 error, perhaps confusing N_k=45 with 45/1000.

Eliminate the wrong options

Which statement about EM for GMM is TRUE?

3 of these 4 are wrong. Strike them one at a time, and say what rules each one out before you strike the next. The survivor is the answer.

• A. EM monotonically increases the log-likelihood but may converge to a local maximum
• B. EM converges to the global maximum because the GMM log-likelihood is convex
• C. EM can decrease the log-likelihood on some steps if sigma_k becomes very small
• D. EM is guaranteed to find the same solution regardless of initialization

Survives elimination: A

Why: EM's convergence theorem (Dempster, Laird, Rubin 1977) guarantees log-likelihood is non-decreasing at every step. However, the GMM log-likelihood is non-convex, so EM can stop at a local maximum. Multiple restarts are the standard mitigation.

Check your understanding

Which statement about EM for GMM is TRUE?

• A. EM monotonically increases the log-likelihood but may converge to a local maximum (correct)
• B. EM converges to the global maximum because the GMM log-likelihood is convex
• C. EM can decrease the log-likelihood on some steps if sigma_k becomes very small
• D. EM is guaranteed to find the same solution regardless of initialization

Answer: A

Why: EM's convergence theorem (Dempster, Laird, Rubin 1977) guarantees log-likelihood is non-decreasing at every step. However, the GMM log-likelihood is non-convex, so EM can stop at a local maximum. Multiple restarts are the standard mitigation.

Why B tempts people

The GMM log-likelihood is explicitly non-convex in (pi, mu, Sigma) — there is no global-maximum guarantee from EM alone.

Why C tempts people

A degenerate sigma (near zero) can cause numerical overflow of the Gaussian PDF, but the algorithm does NOT intentionally decrease the LL — this is a numerical failure mode, not an intended step.

Why D tempts people

Different initializations converge to different local optima; EM is highly initialization-sensitive, which is why n_init > 1 is recommended.

Discussion prompt

Implement GMM EM from scratch in NumPy/SciPy, validate it against sklearn, then use it for anomaly detection on the iris dataset.

That is stated as though it always holds. Do one of two things: produce a case where it fails, or say precisely what rules such a case out. "It just does" is not on the menu.

Hint: Hunt at the extremes first — zero, one, negative, empty, equal. If every extreme survives, the reason they survive is the proof.

Comparison matrix

From Milestone 3 — 20-iteration loop & validation: every row here is a choice with a cost. Fill the mean 0 column, then say which row you would actually pick and what you give up for it.

Comparison matrix

From Milestone 4 — anomaly detection on iris: refill the value column from what you know. The rest of the table is as it appeared.

Draw it

One page, no notation unless you need it: draw how these connect — The GMM density model · E-step — soft responsibilities · M-step — update parameters · GMM from scratch + applications · Your turn: build GMM from scratch. Put an arrow wherever one of them is what makes another possible, and label the arrow with why.