Lesson 69: kNN and the Curse of Dimensionality

Discussion prompt

Before we open Lesson 69: kNN and the Curse of Dimensionality: without looking back, what was the main idea of AdaBoost, and what could you do by the end of it that you could not do before?

Hint: One sentence for the idea, one for the skill. If the second one is blank, that is the part to revisit.

Discussion prompt

Think of k=5 as asking five of your nearest neighbors to vote. If four say 'spam' and one says 'ham', you go with spam. The noise in any single neighbor averages out over the group.

That is stated as though it always holds. Do one of two things: produce a case where it fails, or say precisely what rules such a case out. "It just does" is not on the menu.

Hint: Hunt at the extremes first — zero, one, negative, empty, equal. If every extreme survives, the reason they survive is the proof.

Predict first

Classify query (2.5, 2.5) using k=3 against 5 labeled training points. Predict the answer before computing.

Commit before you compute: what does kNN from scratch — 5-point trace come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: Sort distances: indices [1,2,0,4,3]; k=3 neighbors are classes [0,1,0] -> majority class 0

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. Two class-0 votes beat one class-1 vote: prediction is 0.

Comparison matrix

From kNN from scratch — 5-point trace: refill the point column from what you know. The rest of the table is as it appeared.

Comparison matrix

From L2, L1, Minkowski, cosine, Mahalanobis: every row here is a choice with a cost. Fill the formula column, then say which row you would actually pick and what you give up for it.

Discussion prompt

Compute L2, L1, and cosine distance from A=(1,2) to B=(4,6) and to C=(1,3). Which is closer under each metric?

What do you need to know — or decide — before the first line can be written? List everything the problem has to hand you.

Hint: Anything you would have to invent to get started is a thing the problem must supply.

Step through it

Step through Distance metric worked example one row at a time. What is driving the change, and what would the row after the last one be?

1. Step 1: metric is L2
2. Step 2: metric is L1
3. Step 3: metric is cosine

Predict first

A student writes this, and it looks reasonable:

Features: age in years (range 0-90) and income_normalized (range 0-1). Feed both directly to L2 kNN.

It is wrong. Say what breaks — and say it before you turn the page.

Correct: A difference of 50 in age contributes 2500 to squared L2, while the entire income range contributes at most 1.

Apply StandardScaler before kNN whenever features differ in scale.

Why: A difference of 50 in age contributes 2500 to squared L2, while the entire income range contributes at most 1. kNN effectively ignores income entirely.

Discussion prompt

The rule this trap just fixed:

Z-score normalization gives every feature mean 0 and std 1. This is mandatory for distance-based models including kNN, SVM (L15), and clustering (L52).

Now break it on purpose. Build a case that violates it and follow the consequences until something visibly fails. Where does the failure first show up — and would you have noticed it if you had not been looking?

Hint: The dangerous rules are the ones whose violation still produces an answer. If yours fails loudly, try to find one that fails quietly.

Discussion prompt

Explain In high d, all points become equidistant by analogy to something with no Machine Learning in it at all — a queue, a recipe, a map, a bank balance, whatever fits. Then say where your analogy breaks.

Hint: An analogy that never breaks is not an analogy, it is the same idea wearing a hat. Find the seam — that is the part that is actually new.

Predict first

Draw 1000 standard-normal points in d dimensions, then measure max/min distance to a query. Watch the ratio collapse as d increases.

Commit before you compute: what does Curse of dimensionality — measured come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: d=2 ratio 60.4; d=10 ratio 3.6; d=50 ratio 1.8; d=100 ratio 1.4; d=500 ratio 1.2

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. Even at d=500 the ratio is only 1.19 -- the nearest and farthest neighbor are 19% apart in distance.

Step through it

Step through Curse of dimensionality — measured one row at a time. What is driving the change, and what would the row after the last one be?

1. Step 1: dim is 2
2. Step 2: dim is 10
3. Step 3: dim is 50
4. Step 4: dim is 100
5. Step 5: dim is 500

Sort into buckets

These are the pieces of Lesson 69: kNN and the Curse of Dimensionality, out of order. Put each one back under the part of the lesson it belongs to.

kNN: majority-vote classification

The algorithm: no training, O(n) prediction; Why majority vote?; kNN from scratch — 5-point trace

Distance metrics

L2, L1, Minkowski, cosine, Mahalanobis; Distance metric worked example

Curse of dimensionality

In high d, all points become equidistant; Curse of dimensionality — measured; Practical consequence: when to avoid kNN

s1

kNN: majority-vote classification is where Lesson 69: kNN and the Curse of Dimensionality puts The algorithm: no training, O(n) prediction, Why majority vote?, kNN from scratch — 5-point trace. Knowing which part of the lesson a problem belongs to is most of knowing which method to reach for.

s2

Distance metrics is where Lesson 69: kNN and the Curse of Dimensionality puts L2, L1, Minkowski, cosine, Mahalanobis, Distance metric worked example. Knowing which part of the lesson a problem belongs to is most of knowing which method to reach for.

s3

Curse of dimensionality is where Lesson 69: kNN and the Curse of Dimensionality puts In high d, all points become equidistant, Curse of dimensionality — measured, Practical consequence: when to avoid kNN. Knowing which part of the lesson a problem belongs to is most of knowing which method to reach for.

Sort into buckets

Sort these by exact?, from memory, without looking back at KD-tree, ball tree, and approximate NN. Telling them apart on the spot is the skill; the table is only where the answer happens to be written down.

yes

brute force; KD-tree; ball tree

approx

FAISS / HNSW

g1

exact? is "yes" for brute force, KD-tree, ball tree — that is what the table on "KD-tree, ball tree, and approximate NN" records, and it is the single property separating this group from the rest.

g2

exact? is "approx" for FAISS / HNSW — that is what the table on "KD-tree, ball tree, and approximate NN" records, and it is the single property separating this group from the rest.

Discussion prompt

Compare predict-time for brute force, KD-tree, and ball tree on n=5000, d=10 classification (make_classification). Same accuracy expected.

What do you need to know — or decide — before the first line can be written? List everything the problem has to hand you.

Hint: Anything you would have to invent to get started is a thing the problem must supply.

Step through it

Step through KD-tree vs brute force — timing on 5000 points one row at a time. One of these columns never changes — find it, and say why it cannot.

1. Step 1: algorithm is brute
2. Step 2: algorithm is kd_tree
3. Step 3: algorithm is ball_tree

Discussion prompt

Explain Weighted kNN: 1/distance votes to a student a year behind you. No notation, no jargon they have not met — and it still has to be true.

Hint: If your explanation needs a symbol they have never seen, you are describing the notation rather than the idea.

Predict first

Re-classify query (2.5, 2.5) with k=3 1/distance weights. The two class-0 neighbors are equidistant from the class-1 neighbor — so weighted vote changes nothing here, but the math is the point.

Commit before you compute: what does Weighted kNN trace — same 5-point example come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: Weights [0.6325, 0.6325, 0.4714]; class-0 total 1.1039, class-1 total 0.6325 -> prediction 0

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. Neighbors 1 and 2 are equidistant (1.5811) so equal weight; neighbor 0 is farther (2.1213) so lower weight 0.4714.

Comparison matrix

From Weighted kNN trace — same 5-point example: refill the class column from what you know. The rest of the table is as it appeared.

Discussion prompt

Explain Bias-variance tradeoff over k by analogy to something with no Machine Learning in it at all — a queue, a recipe, a map, a bank balance, whatever fits. Then say where your analogy breaks.

Hint: An analogy that never breaks is not an analogy, it is the same idea wearing a hat. Find the seam — that is the part that is actually new.

Predict first

A student writes this, and it looks reasonable:

k=1 achieves 1.0000 training accuracy on every dataset — it memorizes perfectly. Use k=1 as the default.

It is wrong. Say what breaks — and say it before you turn the page.

Correct: k=1 always classifies every training point correctly (it is its own nearest neighbor).

Evaluate k on a held-out validation set (or cross-validation), never on the training set.

Why: k=1 always classifies every training point correctly (it is its own nearest neighbor). Training accuracy is a useless signal here -- it is always 1.0 regardless of data.

Sort into buckets

Some of these hold up and some are the exact mistakes this lesson is built to prevent. Sort them.

Holds up

Raw image pixels (d=784 for MNIST), raw text TF-IDF vectors (d=10k+), and unfiltered sensor arrays all violate the low-d assumption kNN requires.; Standard kNN gives every neighbor one vote. Weighted kNN gives each neighbor a vote proportional to 1/distance — a very close neighbor counts more than a distant one.; Rule of thumb: tune k via cross-validation. Common starting points are k=sqrt(n) or k=5. Always an odd k for binary classification to avoid ties.

Breaks

Features: age in years (range 0-90) and income_normalized (range 0-1). Feed both directly to L2 kNN.; k=1 achieves 1.0000 training accuracy on every dataset — it memorizes perfectly. Use k=1 as the default.

sound

These are stated as this lesson states them — each one survives the edge cases Lesson 69: kNN and the Curse of Dimensionality puts it through.

flawed

Each of these is lifted from a trap in this deck: reasonable-sounding, and wrong in a way that only shows up once you rely on it.

Put in order

These are the steps of kNN decision recipe, scrambled. Put them back in order before the next slide shows you.

1. Scale features with StandardScaler before any distance computation
2. Check dimensionality: if d > 20, reduce with PCA first
3. Choose metric: L2 (default), cosine (embeddings/text), L1 (sparse/outlier-robust)
4. Tune k via cross-validation; start at k=5 or k=sqrt(n), always odd for binary
5. Choose algorithm: kd_tree for d<20; ball_tree for d<100; brute or FAISS for d>100
6. Weighted kNN (weights='distance') when neighbors vary greatly in proximity

Why: This is the order the recipe itself gives. Recalling the sequence without the slide in front of you is the difference between recognising the method and being able to run it — most of what goes wrong in practice is a step done out of turn.

Discussion prompt

kNN decision recipe works on the cases you have just seen. Push it to the edge: what is the most degenerate input it still handles — empty, zero, one item, everything equal — and what is the first case where it stops being true? Name the case, not just "it breaks".

Hint: Try the smallest legal input, then the largest, then the one where two things collide. Methods are specified at their edges; the middle takes care of itself.

Eliminate the wrong options

As the number of dimensions d grows, what happens to the ratio max_distance / min_distance among n random points?

3 of these 4 are wrong. Strike them one at a time, and say what rules each one out before you strike the next. The survivor is the answer.

• A. It approaches 1 — all points become roughly equidistant
• B. It grows without bound — the farthest point gets much farther
• C. It stays constant because distances all scale with sqrt(d)
• D. It approaches 0 — all points collapse to the origin

Survives elimination: A

Why: Distance concentrates around the mean in high d. In our experiment: d=2 gives ratio 60.4; d=10 gives 3.6; d=500 gives 1.18. The gap between nearest and farthest neighbor vanishes, making kNN's ranking meaningless.

Check your understanding

As the number of dimensions d grows, what happens to the ratio max_distance / min_distance among n random points?

• A. It approaches 1 — all points become roughly equidistant (correct)
• B. It grows without bound — the farthest point gets much farther
• C. It stays constant because distances all scale with sqrt(d)
• D. It approaches 0 — all points collapse to the origin

Answer: A

Why: Distance concentrates around the mean in high d. In our experiment: d=2 gives ratio 60.4; d=10 gives 3.6; d=500 gives 1.18. The gap between nearest and farthest neighbor vanishes, making kNN's ranking meaningless.

Why B tempts people

Both max and min distance scale with sqrt(d) — they grow, but at the same rate, so their ratio shrinks rather than grows.

Why C tempts people

The mean distance does scale with sqrt(d), but max and min concentrate tightly around that mean; the ratio (max/min) narrows, not stays flat.

Why D tempts people

Points do not collapse to the origin in L2 norm — they spread outward with sqrt(d). The origin has no special role.

Predict first

You have n=1M vectors in d=128 dimensions (e.g. image embeddings). Which nearest-neighbor strategy is best?

Answer it in your own words, now, with nothing to choose from. The options are on the next slide — and picking the right one off a list is an easier skill than producing it.

Correct: FAISS or HNSW approximate nearest neighbor

Why: At d=128, KD-tree degrades to near-brute-force (it only wins for d<20). Brute-force over 1M vectors is infeasible at inference time. FAISS/HNSW gives sub-linear approximate search that is fast enough and accurate enough for embedding retrieval.

Check your understanding

You have n=1M vectors in d=128 dimensions (e.g. image embeddings). Which nearest-neighbor strategy is best?

• A. FAISS or HNSW approximate nearest neighbor (correct)
• B. KD-tree exact search
• C. Brute-force O(nd) scan
• D. Ball tree exact search

Answer: A

Why: At d=128, KD-tree degrades to near-brute-force (it only wins for d<20). Brute-force over 1M vectors is infeasible at inference time. FAISS/HNSW gives sub-linear approximate search that is fast enough and accurate enough for embedding retrieval.

Why B tempts people

KD-tree is efficient only for d < 20; at d=128 it reverts to O(nd) without the overhead of tree construction. It is not recommended above d=20.

Why C tempts people

Brute-force is O(nd) per query: 1M x 128 = 128M multiplications every inference call. This is feasible for batch offline jobs but not real-time retrieval.

Why D tempts people

Ball tree handles curved manifolds better than KD-tree and works to d~100, but at d=128 and n=1M it is still too slow for real-time use; FAISS is the correct tool.

Eliminate the wrong options

Increasing k in kNN (holding everything else constant) has what effect on model bias and variance?

3 of these 4 are wrong. Strike them one at a time, and say what rules each one out before you strike the next. The survivor is the answer.

• A. Increases bias, decreases variance
• B. Decreases bias, increases variance
• C. Increases both bias and variance
• D. Decreases both bias and variance

Survives elimination: A

Why: Large k averages more neighbors, smoothing the decision boundary — this reduces variance (less sensitive to individual points) but introduces bias (the boundary can't capture fine structure). k=1 is maximum variance/minimum bias; k=n is maximum bias/minimum variance.

Check your understanding

Increasing k in kNN (holding everything else constant) has what effect on model bias and variance?

• A. Increases bias, decreases variance (correct)
• B. Decreases bias, increases variance
• C. Increases both bias and variance
• D. Decreases both bias and variance

Answer: A

Why: Large k averages more neighbors, smoothing the decision boundary — this reduces variance (less sensitive to individual points) but introduces bias (the boundary can't capture fine structure). k=1 is maximum variance/minimum bias; k=n is maximum bias/minimum variance.

Why B tempts people

This is backwards: large k smooths the boundary (lower variance, higher bias). Small k memorizes (high variance, low bias).

Why C tempts people

You cannot simultaneously increase both: the bias-variance tradeoff is a tradeoff. Smoothing always trades variance for bias.

Why D tempts people

Increasing k does reduce variance, but it does so at the cost of increased bias. There is no free lunch — bias must rise when variance falls.

Discussion prompt

Implement kNN from scratch supporting both L2 and cosine distance. Benchmark it against sklearn on iris, then demonstrate the curse of dimensionality.

That is stated as though it always holds. Do one of two things: produce a case where it fails, or say precisely what rules such a case out. "It just does" is not on the menu.

Hint: Hunt at the extremes first — zero, one, negative, empty, equal. If every extreme survives, the reason they survive is the proof.

Comparison matrix

From Milestone 2 — curse of dimensionality: every row here is a choice with a cost. Fill the ratio max/min column, then say which row you would actually pick and what you give up for it.

Comparison matrix

From The full program: refill the value column from what you know. The rest of the table is as it appeared.

Draw it

One page, no notation unless you need it: draw how these connect — kNN: majority-vote classification · Distance metrics · Curse of dimensionality · Efficient search & weighted kNN · Your turn: kNN from scratch. Put an arrow wherever one of them is what makes another possible, and label the arrow with why.