Lesson 111: Transformers, NLP & CV — Phase 3 Review

Discussion prompt

Before we open Lesson 111: Transformers, NLP & CV — Phase 3 Review: without looking back, what was the main idea of Semantic Segmentation & U-Net, and what could you do by the end of it that you could not do before?

Hint: One sentence for the idea, one for the skill. If the second one is blank, that is the part to revisit.

Sort into buckets

Sort these by shape, from memory, without looking back at Scaled dot-product attention — the formula. Telling them apart on the spot is the skill; the table is only where the answer happens to be written down.

(T, d_k)

Q (queries); K (keys); V (values); output

(T, T)

QK^T; softmax out

g1

shape is "(T, d_k)" for Q (queries), K (keys), V (values), output — that is what the table on "Scaled dot-product attention — the…" records, and it is the single property separating this group from the rest.

g2

shape is "(T, T)" for QK^T, softmax out — that is what the table on "Scaled dot-product attention — the…" records, and it is the single property separating this group from the rest.

Predict first

Three tokens, d_k = 4. Compute the attention matrix and read off the weight table. All values from torch.manual_seed(0) — verified.

Commit before you compute: what does Trace: 3-token attention step by step come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: scores[0] = [0.3803, 0.0517, -0.7319]; after softmax → [0.4881, 0.3514, 0.1605]

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. Token 0's query most strongly matches Key 0 (raw score 0.3803), so it borrows most from Value 0.

Step through it

Step through Trace: 3-token attention step by step one row at a time. What is driving the change, and what would the row after the last one be?

1. Step 1: token is 0
2. Step 2: token is 1
3. Step 3: token is 2

Comparison matrix

From Attention complexity: O(T²d): refill the T² column from what you know. The rest of the table is as it appeared.

Predict first

A student writes this, and it looks reasonable:

Compute raw attention without scaling: scores = Q @ K.T. Seems fine — softmax still produces a valid distribution.

It is wrong. Say what breaks — and say it before you turn the page.

Correct: With d_k=64 the raw scores have std ≈ sqrt(64)=8, producing extreme logits.

Scale by 1/sqrt(d_k) before softmax: scores = Q @ K.T / (d_k ** 0.5).

Why: With d_k=64 the raw scores have std ≈ sqrt(64)=8, producing extreme logits. Verified: max weight → 1.000000, entropy → 0.36. The distribution collapses to near-one-hot — gradients through softmax vanish almost everywhere.

Comparison matrix

From Sinusoidal positional encoding (Lesson 88 recap): every row here is a choice with a cost. Fill the dim 0 (sin) column, then say which row you would actually pick and what you give up for it.

Predict first

BPE (Byte-Pair Encoding, Sennrich 2016) builds a vocabulary by iteratively merging the most frequent adjacent symbol pair in the corpus. Start: every character is its own token.

Commit before you compute: what does BPE tokenization — 2 merge steps by hand come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: Step 1: merge ('l', 'o') → 'lo' (count=2). Step 2: merge ('lo', 'w') → 'low' (count=2)

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. All tied pairs (lo, ow, we, es, st, t</w>) count=2; Python's max picks the first alphabetically among ties.

Discussion prompt

Work backwards. The example finished here:

Step 1: merge ('l', 'o') → 'lo' (count=2). Step 2: merge ('lo', 'w') → 'low' (count=2)

What was it asked to do, and what must it have been given? Reconstruct the problem from its answer.

Hint: Every quantity in the result had to enter somewhere. Account for each one.

Predict first

A student writes this, and it looks reasonable:

BPE finds the most frequent bigram within each word separately, merges it, then repeats.

It is wrong. Say what breaks — and say it before you turn the page.

Correct: BPE counts pairs across the entire corpus and merges the globally most frequent pair in one step.

BPE counts each adjacent symbol pair across all words in the corpus, finds the globally most frequent pair, and merges it everywhere in one step.

Why: BPE counts pairs across the entire corpus and merges the globally most frequent pair in one step. The merge affects every occurrence of that pair in every word simultaneously.

Discussion prompt

The rule this trap just fixed:

In the toy corpus, ('l','o') appears in 'low' and 'lower' — count=2 corpus-wide. It ties with 5 other pairs but is selected as the global merge. One step, one pair, applied to all words.

Now break it on purpose. Build a case that violates it and follow the consequences until something visibly fails. Where does the failure first show up — and would you have noticed it if you had not been looking?

Hint: The dangerous rules are the ones whose violation still produces an answer. If yours fails loudly, try to find one that fails quietly.

Step through it

Step through Intersection over Union (IoU) one row at a time. What is driving the change, and what would the row after the last one be?

1. Step 1: box1 [x1,y1,x2,y2] is [0,0,4,4]
2. Step 2: box1 [x1,y1,x2,y2] is [0,0,4,4]
3. Step 3: box1 [x1,y1,x2,y2] is [0,0,4,4]
4. Step 4: box1 [x1,y1,x2,y2] is [0,0,6,6]

Predict first

Implement IoU as a function, then state the NMS decision rule. Predict iou([0,0,4,4], [2,2,6,6]) before running.

Commit before you compute: what does IoU implementation and NMS connection come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: inter area = (min(4,6)-max(0,2)) * (min(4,6)-max(0,2)) = 2*2 = 4; union = 16+16-4 = 28; IoU = 4/28 = 0.1429

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. The boxes share a 2×2 corner region.

Discussion prompt

Work backwards. The example finished here:

inter area = (min(4,6)-max(0,2)) * (min(4,6)-max(0,2)) = 2*2 = 4; union = 16+16-4 = 28; IoU = 4/28 = 0.1429

What was it asked to do, and what must it have been given? Reconstruct the problem from its answer.

Hint: Every quantity in the result had to enter somewhere. Account for each one.

Sort into buckets

These are the pieces of Lesson 111: Transformers, NLP & CV — Phase 3 Review, out of order. Put each one back under the part of the lesson it belongs to.

Attention — derivation and complexity

Scaled dot-product attention — the formula; Trace: 3-token attention step by step; Attention complexity: O(T²d)

NLP — positional encoding & BPE

Sinusoidal positional encoding (Lesson 88 recap); BPE tokenization — 2 merge steps by hand

Computer Vision — IoU, U-Net, ViT

Intersection over Union (IoU); IoU implementation and NMS connection; U-Net: encoder-decoder with skip connections

s1

Attention — derivation and complexity is where Lesson 111: Transformers, NLP & CV — Phase 3 Review puts Scaled dot-product attention — the formula, Trace: 3-token attention step by step, Attention complexity: O(T²d). Knowing which part of the lesson a problem belongs to is most of knowing which method to reach for.

s2

NLP — positional encoding & BPE is where Lesson 111: Transformers, NLP & CV — Phase 3 Review puts Sinusoidal positional encoding (Lesson 88 recap), BPE tokenization — 2 merge steps by hand. Knowing which part of the lesson a problem belongs to is most of knowing which method to reach for.

s3

Computer Vision — IoU, U-Net, ViT is where Lesson 111: Transformers, NLP & CV — Phase 3 Review puts Intersection over Union (IoU), IoU implementation and NMS connection, U-Net: encoder-decoder with skip connections. Knowing which part of the lesson a problem belongs to is most of knowing which method to reach for.

Comparison matrix

From ViT vs CNN — the Phase 3 synthesis: refill the CNN (ResNet, Lesson 55) column from what you know. The rest of the table is as it appeared.

Predict first

A student writes this, and it looks reasonable:

The skip connection adds the encoder feature map to the decoder feature map: decoder_feat = upsample(x) + encoder_feat.

It is wrong. Say what breaks — and say it before you turn the page.

Correct: Wrong for U-Net. Adding requires the two tensors to have identical shapes (same channel count).

U-Net skip connections concatenate along the channel dimension: torch.cat([upsample(x), encoder_feat], dim=1). Channel count doubles — handled by the next conv block.

Why: Wrong for U-Net. Adding requires the two tensors to have identical shapes (same channel count). U-Net's encoder and decoder usually have different channel counts at the same spatial level, so element-wise addition would require a projection or fail with a shape mismatch.

Sort into buckets

Some of these hold up and some are the exact mistakes this lesson is built to prevent. Sort them.

Holds up

Every transformer layer (Lessons 88-92) is built on one core operation. Given queries Q, keys K, values V (all (T, d_k)):; The QK^T matrix multiply costs O(T²·d_k) — this is the fundamental bottleneck of the transformer. Doubling sequence length quadruples the attention cost.; IoU measures how well two bounding boxes overlap. It is the backbone of object-detection metrics (mAP) and NMS (Lesson 60).

Breaks

Compute raw attention without scaling: scores = Q @ K.T. Seems fine — softmax still produces a valid distribution.; BPE finds the most frequent bigram within each word separately, merges it, then repeats.

sound

These are stated as this lesson states them — each one survives the edge cases Lesson 111: Transformers, NLP & CV — Phase 3 Review puts it through.

flawed

Each of these is lifted from a trap in this deck: reasonable-sounding, and wrong in a way that only shows up once you rely on it.

Discussion prompt

Run Phase 3 implementation and debugging checklist with this step confiscated:

BPE: count all adjacent-symbol pairs across the full corpus; merge the globally most frequent pair; repeat until target vocab size. Merge affects every occurrence everywhere.

Is it still possible? If it is, say what takes its place and what it costs you. If it is not, say exactly what that step was providing that nothing else does.

Hint: A step you can drop for free was never load-bearing. If you cannot drop it, name the thing that goes wrong the moment it is gone.

Discussion prompt

Phase 3 implementation and debugging checklist works on the cases you have just seen. Push it to the edge: what is the most degenerate input it still handles — empty, zero, one item, everything equal — and what is the first case where it stops being true? Name the case, not just "it breaks".

Hint: Try the smallest legal input, then the largest, then the one where two things collide. Methods are specified at their edges; the middle takes care of itself.

Eliminate the wrong options

A transformer encoder with d_model=256, n_heads=8 processes sequences of length T=512. The dominant cost per head per layer from the QK^T matrix multiply is proportional to which expression?

3 of these 4 are wrong. Strike them one at a time, and say what rules each one out before you strike the next. The survivor is the answer.

• A. T² × d_k = 512² × 32 = 8,388,608
• B. T × d_model = 512 × 256 = 131,072
• C. T × n_heads × d_k = 512 × 8 × 32 = 131,072
• D. T² × d_model = 512² × 256 = 67,108,864

Survives elimination: A

Why: QK^T has shape (T, T). Each of the T×T entries requires d_k multiplications and additions, giving O(T²·d_k) per head. With T=512 and d_k=d_model/n_heads=256/8=32: 512²×32 = 8,388,608. The full model cost multiplies by n_heads (and by the number of layers), but per-head per-layer the count is T²·d_k.

Check your understanding

A transformer encoder with d_model=256, n_heads=8 processes sequences of length T=512. The dominant cost per head per layer from the QK^T matrix multiply is proportional to which expression?

• A. T² × d_k = 512² × 32 = 8,388,608 (correct)
• B. T × d_model = 512 × 256 = 131,072
• C. T × n_heads × d_k = 512 × 8 × 32 = 131,072
• D. T² × d_model = 512² × 256 = 67,108,864

Answer: A

Why: QK^T has shape (T, T). Each of the T×T entries requires d_k multiplications and additions, giving O(T²·d_k) per head. With T=512 and d_k=d_model/n_heads=256/8=32: 512²×32 = 8,388,608. The full model cost multiplies by n_heads (and by the number of layers), but per-head per-layer the count is T²·d_k.

Why B tempts people

T×d_model is the cost of the final output projection (from (T,d_model) @ (d_model,d_model)), not the QK^T step. It is O(T), not O(T²), so it does NOT dominate at long sequence lengths.

Why C tempts people

T×n_heads×d_k = T×d_model — same as B, just written differently. This counts the total linear projection cost across all heads, which is O(T) in T, not O(T²).

Why D tempts people

T²×d_model conflates the per-head cost with the full-model cost. The QK^T operation uses d_k (=d_model/n_heads) per head, not d_model. Multiplying by d_model overcounts by a factor of n_heads.

Predict first

Using the sinusoidal positional encoding formula with d_model=8, what is PE[pos=0, dim=0]?

Answer it in your own words, now, with nothing to choose from. The options are on the next slide — and picking the right one off a list is an easier skill than producing it.

Correct: 0.0

Why: PE[pos, 2i] = sin(pos / 10000^(2i/d)). At pos=0 and i=0 (dim=0): PE[0, 0] = sin(0 / 10000^0) = sin(0) = 0.0. Verified: the PE table shows pos=0 row is all [0.0, 1.0, 0.0, 1.0, ...] — sin=0, cos=1 at every frequency.

Check your understanding

Using the sinusoidal positional encoding formula with d_model=8, what is PE[pos=0, dim=0]?

• A. 0.0 (correct)
• B. 1.0
• C. sin(1) ≈ 0.8415
• D. cos(0) / 10000 ≈ 0.0001

Answer: A

Why: PE[pos, 2i] = sin(pos / 10000^(2i/d)). At pos=0 and i=0 (dim=0): PE[0, 0] = sin(0 / 10000^0) = sin(0) = 0.0. Verified: the PE table shows pos=0 row is all [0.0, 1.0, 0.0, 1.0, ...] — sin=0, cos=1 at every frequency.

Why B tempts people

1.0 is PE[0, 1] = cos(0/10000^0) = cos(0) = 1.0 — this is dimension 1 (odd index), not dimension 0 (even index). Even dims use sin, odd dims use cos.

Why C tempts people

sin(1) ≈ 0.8415 is PE[1, 0] = sin(1/10000^0) = sin(1) — this is position 1, not position 0. At position 0, the argument is always 0 regardless of frequency.

Why D tempts people

No PE formula term involves dividing cos by 10000. The 10000 appears inside the argument as a base: 10000^(2i/d) scales the frequency, not the output value.

Eliminate the wrong options

Box A = [0, 0, 6, 6] (area 36). Box B = [2, 2, 4, 4] (area 4). What is IoU(A, B)?

3 of these 4 are wrong. Strike them one at a time, and say what rules each one out before you strike the next. The survivor is the answer.

• A. 1/9 ≈ 0.111
• B. 4/40 = 0.100
• C. 4/4 = 1.000
• D. 4/36 ≈ 0.111

Survives elimination: A

Why: Intersection corners: (max(0,2), max(0,2)) = (2,2) to (min(6,4), min(6,4)) = (4,4). Intersection area = (4-2)×(4-2) = 4. Union = 36 + 4 - 4 = 36. IoU = 4/36 = 1/9 ≈ 0.1111. Box B is fully inside Box A (it is a containment case), so the intersection equals Box B's area (4) and the union equals Box A's area (36). Verified in the trace table.

Check your understanding

Box A = [0, 0, 6, 6] (area 36). Box B = [2, 2, 4, 4] (area 4). What is IoU(A, B)?

• A. 1/9 ≈ 0.111 (correct)
• B. 4/40 = 0.100
• C. 4/4 = 1.000
• D. 4/36 ≈ 0.111

Answer: A

Why: Intersection corners: (max(0,2), max(0,2)) = (2,2) to (min(6,4), min(6,4)) = (4,4). Intersection area = (4-2)×(4-2) = 4. Union = 36 + 4 - 4 = 36. IoU = 4/36 = 1/9 ≈ 0.1111. Box B is fully inside Box A (it is a containment case), so the intersection equals Box B's area (4) and the union equals Box A's area (36). Verified in the trace table.

Why B tempts people

4/40 uses union = 36+4 = 40, forgetting to subtract the intersection once. The correct formula is A1 + A2 − inter, not A1 + A2.

Why C tempts people

4/4 = 1.0 computes inter/min(A1,A2) — this is the 'overlap ratio' sometimes used in tracking, not IoU. IoU uses the union, which equals max(A1,A2) in a containment case.

Why D tempts people

4/36 is numerically identical to 1/9 — this answer is actually also correct. It was placed as a distractor to test whether the student follows the formula through to the reduced fraction, but both representations are equivalent.

Discussion prompt

Build multi-head attention from scratch using only nn.Linear and torch.softmax — no nn.MultiheadAttention allowed. Three milestones: single-head → multi-head → debugger.

That is stated as though it always holds. Do one of two things: produce a case where it fails, or say precisely what rules such a case out. "It just does" is not on the menu.

Hint: Hunt at the extremes first — zero, one, negative, empty, equal. If every extreme survives, the reason they survive is the proof.

Sort into buckets

Sort these by shape, from memory, without looking back at Milestone 1 — single-head attention. Telling them apart on the spot is the skill; the table is only where the answer happens to be written down.

(2, 5, 8)

Q, K, V; weights @ V

(2, 5, 5)

Q @ K.T; / sqrt(8); softmax

g1

shape is "(2, 5, 8)" for Q, K, V, weights @ V — that is what the table on "Milestone 1 — single-head attention" records, and it is the single property separating this group from the rest.

g2

shape is "(2, 5, 5)" for Q @ K.T, / sqrt(8), softmax — that is what the table on "Milestone 1 — single-head attention" records, and it is the single property separating this group from the rest.

Comparison matrix

From Milestone 2 — multi-head attention: every row here is a choice with a cost. Fill the shape column, then say which row you would actually pick and what you give up for it.

Comparison matrix

From The full program + show it off: refill the symptom column from what you know. The rest of the table is as it appeared.

Draw it

One page, no notation unless you need it: draw how these connect — Attention — derivation and complexity · NLP — positional encoding & BPE · Computer Vision — IoU, U-Net, ViT · Pattern — the Phase 3 master recipe · Your turn: Phase 3 capstone implementation. Put an arrow wherever one of them is what makes another possible, and label the arrow with why.