Lesson 52: k-Means Clustering

Comparison matrix

From The algorithm in two moves: every row here is a choice with a cost. Fill the cost column, then say which row you would actually pick and what you give up for it.

Discussion prompt

WCSS measures total intra-cluster spread. k-Means is a coordinate-descent optimizer on this objective.

That is stated as though it always holds. Do one of two things: produce a case where it fails, or say precisely what rules such a case out. "It just does" is not on the menu.

Hint: Hunt at the extremes first — zero, one, negative, empty, equal. If every extreme survives, the reason they survive is the proof.

Predict first

Six 2D points; initial centroids at [1,1] and [5,7]. Run two iterations by hand.

Commit before you compute: what does Trace: 6 points, 2 centroids, 2 iterations come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: Iter 1: labels = [0,0,0,1,1,1], c0=[1.833,2.333], c1=[4.333,5.667], WCSS=10.6667

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. Points closer to [1,1] join cluster 0; those closer to [5,7] join cluster 1.

Comparison matrix

From Trace: 6 points, 2 centroids, 2 iterations: refill the c0 column from what you know. The rest of the table is as it appeared.

Discussion prompt

Explain Why random init fails by analogy to something with no Machine Learning in it at all — a queue, a recipe, a map, a bank balance, whatever fits. Then say where your analogy breaks.

Hint: An analogy that never breaks is not an analogy, it is the same idea wearing a hat. Find the seam — that is the part that is actually new.

Predict first

Six collinear points at 0,1,5,6,10,11. First seed = [0,0]. Compute the d² probabilities for the second seed.

Commit before you compute: what does k-Means++ selection probabilities come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: x=11 has P=0.4276; x=10 has P=0.3534; x=1 has P=0.0035

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. d² grows quadratically with distance — the seeding rule over-represents outlying points, spreading initial seeds across the data.

Step through it

Step through k-Means++ selection probabilities one row at a time. What is driving the change, and what would the row after the last one be?

1. Step 1: x is 0
2. Step 2: x is 1
3. Step 3: x is 5
4. Step 4: x is 6
5. Step 5: x is 10
6. Step 6: x is 11

Predict first

A student writes this, and it looks reasonable:

Implement k-Means++ by weighting each candidate proportional to ‖x − nearest_c‖ (the plain Euclidean distance).

It is wrong. Say what breaks — and say it before you turn the page.

Correct: The plain distance still favors far points, so it 'looks right' locally — but it underweights extreme outliers relative to the theoretically optimal schedule, degrading the O(log k) approximation guarantee that…

Weight proportional to d² — the squared distance to the nearest already-chosen centroid.

Why: The plain distance still favors far points, so it 'looks right' locally — but it underweights extreme outliers relative to the theoretically optimal schedule, degrading the O(log k) approximation guarantee that k-Means++ provides.

Discussion prompt

The rule this trap just fixed:

The squared-distance weighting is the definition of k-Means++; it provides a O(log k) expected-WCSS approximation guarantee. Plain distance loses this bound.

Now break it on purpose. Build a case that violates it and follow the consequences until something visibly fails. Where does the failure first show up — and would you have noticed it if you had not been looking?

Hint: The dangerous rules are the ones whose violation still produces an answer. If yours fails loudly, try to find one that fails quietly.

Step through it

Step through Elbow method: choosing k one row at a time. What is driving the change, and what would the row after the last one be?

1. Step 1: k is 1
2. Step 2: k is 2
3. Step 3: k is 3
4. Step 4: k is 4
5. Step 5: k is 5
6. Step 6: k is 6
7. Step 7: k is 7

Discussion prompt

Explain Silhouette score by analogy to something with no Machine Learning in it at all — a queue, a recipe, a map, a bank balance, whatever fits. Then say where your analogy breaks.

Hint: An analogy that never breaks is not an analogy, it is the same idea wearing a hat. Find the seam — that is the part that is actually new.

Predict first

Point 0 in the make_blobs dataset is assigned to cluster 1 after k-Means (seed 42). Compute its silhouette score from scratch.

Commit before you compute: what does Silhouette: compute s(i) for one point come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: a=1.4486 (tight cluster), b=15.5553 (far from next cluster), s=0.9069

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. b >> a means the point is deep inside its cluster and far from any other — near-perfect silhouette of 0.91.

Discussion prompt

Work backwards. The example finished here:

a=1.4486 (tight cluster), b=15.5553 (far from next cluster), s=0.9069

What was it asked to do, and what must it have been given? Reconstruct the problem from its answer.

Hint: Every quantity in the result had to enter somewhere. Account for each one.

Comparison matrix

From k-Means as EM: refill the k-Means action column from what you know. The rest of the table is as it appeared.

Predict first

A student writes this, and it looks reasonable:

Apply k-Means (k=2) to a two-moon dataset — it will find 2 clusters.

It is wrong. Say what breaks — and say it before you turn the page.

Correct: k-Means uses Euclidean distance to a centroid — it can only produce convex, roughly spherical clusters.

Use DBSCAN (density-based; no shape assumption) with eps=0.25, min_samples=5.

Why: k-Means uses Euclidean distance to a centroid — it can only produce convex, roughly spherical clusters. The two moons are non-convex and interlocked; k-Means cuts them down the middle and gets 74.5% accuracy.

Sort into buckets

Some of these hold up and some are the exact mistakes this lesson is built to prevent. Sort them.

Holds up

WCSS = within-cluster sum of squares — the objective k-Means greedily decreases each iteration. It cannot increase.; WCSS measures total intra-cluster spread. k-Means is a coordinate-descent optimizer on this objective.; With random initialization, two initial centroids may land in the same true cluster — the algorithm then wastes iterations and often converges to a bad local minimum.

Breaks

Implement k-Means++ by weighting each candidate proportional to ‖x − nearest_c‖ (the plain Euclidean distance).; Apply k-Means (k=2) to a two-moon dataset — it will find 2 clusters.

sound

These are stated as this lesson states them — each one survives the edge cases Lesson 52: k-Means Clustering puts it through.

flawed

Each of these is lifted from a trap in this deck: reasonable-sounding, and wrong in a way that only shows up once you rely on it.

Put in order

These are the steps of k-Means recipe, scrambled. Put them back in order before the next slide shows you.

1. Init: pick k seeds via k-Means++ (d² weighting)
2. E-step: assign each point to nearest centroid — labels = argmin_k ‖x − μk‖²
3. M-step: update each centroid — μk = mean({xi : label=k})
4. Repeat until labels don't change (WCSS only decreases; local minimum guaranteed)
5. Choose k: elbow on WCSS vs k; confirm with silhouette score
6. Sanity-check shape assumption: if clusters are non-spherical, use DBSCAN or GMM

Why: This is the order the recipe itself gives. Recalling the sequence without the slide in front of you is the difference between recognising the method and being able to run it — most of what goes wrong in practice is a step done out of turn.

Discussion prompt

k-Means recipe works on the cases you have just seen. Push it to the edge: what is the most degenerate input it still handles — empty, zero, one item, everything equal — and what is the first case where it stops being true? Name the case, not just "it breaks".

Hint: Try the smallest legal input, then the largest, then the one where two things collide. Methods are specified at their edges; the middle takes care of itself.

Eliminate the wrong options

k-Means is guaranteed to converge. The solution it converges to is:

3 of these 4 are wrong. Strike them one at a time, and say what rules each one out before you strike the next. The survivor is the answer.

• A. a local minimum of WCSS — not necessarily the global minimum
• B. the global minimum of WCSS — guaranteed by the E/M alternation
• C. a saddle point of the log-likelihood
• D. the global minimum only when initialized with k-Means++

Survives elimination: A

Why: Each E-step and M-step can only decrease or maintain WCSS, so the sequence of WCSS values converges. But the algorithm is coordinate descent on a non-convex objective — the limit depends on initialization and is in general a local minimum. Our random-init run hit WCSS 5693 vs the global 567.

Check your understanding

k-Means is guaranteed to converge. The solution it converges to is:

• A. a local minimum of WCSS — not necessarily the global minimum (correct)
• B. the global minimum of WCSS — guaranteed by the E/M alternation
• C. a saddle point of the log-likelihood
• D. the global minimum only when initialized with k-Means++

Answer: A

Why: Each E-step and M-step can only decrease or maintain WCSS, so the sequence of WCSS values converges. But the algorithm is coordinate descent on a non-convex objective — the limit depends on initialization and is in general a local minimum. Our random-init run hit WCSS 5693 vs the global 567.

Why B tempts people

The WCSS surface is non-convex for k>1; coordinate descent on a non-convex function cannot guarantee the global minimum.

Why C tempts people

k-Means minimizes WCSS, not the log-likelihood (that's EM/GMM). k-Means has no log-likelihood objective.

Why D tempts people

k-Means++ gives a better expected initialization (O(log k) bound) but does not turn the algorithm into a global optimizer — it still converges to a local minimum.

Predict first

Point i has a(i)=1.45 (mean intra-cluster distance) and b(i)=15.56 (nearest-cluster mean distance). Its silhouette score s(i) is closest to:

Answer it in your own words, now, with nothing to choose from. The options are on the next slide — and picking the right one off a list is an easier skill than producing it.

Correct: 0.91

Why: s(i) = (b−a)/max(a,b) = (15.56−1.45)/max(1.45,15.56) = 14.11/15.56 ≈ 0.907 ≈ 0.91. The point is well inside its cluster and far from the next — high positive score.

Check your understanding

Point i has a(i)=1.45 (mean intra-cluster distance) and b(i)=15.56 (nearest-cluster mean distance). Its silhouette score s(i) is closest to:

• A. 0.91 (correct)
• B. 0.09
• C. −0.91
• D. 14.11

Answer: A

Why: s(i) = (b−a)/max(a,b) = (15.56−1.45)/max(1.45,15.56) = 14.11/15.56 ≈ 0.907 ≈ 0.91. The point is well inside its cluster and far from the next — high positive score.

Why B tempts people

0.09 = a/b, not (b−a)/max(a,b). Confusing the ratio direction.

Why C tempts people

Negative silhouette means a > b (point is closer to a different cluster than its own). Here b >> a, so the score is strongly positive.

Why D tempts people

14.11 = b−a is the numerator, not the normalized score. Forgetting to divide by max(a,b).

Eliminate the wrong options

Soft k-Means differs from hard k-Means in that it:

3 of these 4 are wrong. Strike them one at a time, and say what rules each one out before you strike the next. The survivor is the answer.

• A. assigns fractional cluster memberships (responsibilities) instead of 0/1 hard labels
• B. uses Manhattan distance instead of Euclidean distance
• C. allows k to change during training
• D. guarantees convergence to the global optimum

Survives elimination: A

Why: In the EM view of k-Means, the E-step assigns a responsibility r_ik ∈ {0,1}. Soft k-Means relaxes this to r_ik ∈ [0,1] via a softmax — recovering the GMM with equal spherical covariances. Both still alternate E and M steps.

Check your understanding

Soft k-Means differs from hard k-Means in that it:

• A. assigns fractional cluster memberships (responsibilities) instead of 0/1 hard labels (correct)
• B. uses Manhattan distance instead of Euclidean distance
• C. allows k to change during training
• D. guarantees convergence to the global optimum

Answer: A

Why: In the EM view of k-Means, the E-step assigns a responsibility r_ik ∈ {0,1}. Soft k-Means relaxes this to r_ik ∈ [0,1] via a softmax — recovering the GMM with equal spherical covariances. Both still alternate E and M steps.

Why B tempts people

Both hard and soft k-Means use squared Euclidean distance in the standard formulation. Manhattan distance is a separate variant.

Why C tempts people

k is fixed in both formulations. Algorithms that adapt k (like X-means or DBSCAN) are separate.

Why D tempts people

Neither hard nor soft k-Means guarantees the global optimum — both are coordinate-descent methods on a non-convex objective.

Discussion prompt

Build k-Means with both random and k++ initialization on make_blobs(centers=3). Implement the elbow method and silhouette from scratch, then demonstrate k-Means failing on moons.

That is stated as though it always holds. Do one of two things: produce a case where it fails, or say precisely what rules such a case out. "It just does" is not on the menu.

Hint: Hunt at the extremes first — zero, one, negative, empty, equal. If every extreme survives, the reason they survive is the proof.

Comparison matrix

From Milestone 3 — silhouette from scratch: every row here is a choice with a cost. Fill the mean silhouette column, then say which row you would actually pick and what you give up for it.

Comparison matrix

From The full program: refill the value column from what you know. The rest of the table is as it appeared.

Draw it

One page, no notation unless you need it: draw how these connect — k-Means: assign & update · k-Means++ initialization · Elbow & silhouette · Your turn: build k-Means from scratch. Put an arrow wherever one of them is what makes another possible, and label the arrow with why.