Lesson 115: Transformer End-to-End Review

Discussion prompt

Before we open Lesson 115: Transformer End-to-End Review: without looking back, what was the main idea of Phase 3 Mock Exam — Transformers, NLP, and Computer Vision, and what could you do by the end of it that you could not do before?

Hint: One sentence for the idea, one for the skill. If the second one is blank, that is the part to revisit.

Comparison matrix

From Scaled dot-product attention: refill the Shape column from what you know. The rest of the table is as it appeared.

Predict first

The table runs: 1 | QKᵀ (pre-scale) | (4,4) | raw dot products · 2 | ÷ √8 = 0.3536 | (4,4) | scores dampened · 3 | softmax(row) | (4,4) | [0.1942, 0.4102, 0.3242, 0.0714]

In Trace: attention weights with seq=4, d_k=8, given the rows so far: what is the next one — the row where Step is 4?

Correct: 4 | × V | (4,8) | weighted value sum

Why: The relationship between the columns, not the individual numbers, is what generates the next row. Small concrete tensors let us print every number; seed 42 is reproducible.

Comparison matrix

From Trace: attention weights with seq=4, d_k=8: every row here is a choice with a cost. Fill the Values (row 0) column, then say which row you would actually pick and what you give up for it.

Predict first

Lines 9-11: the reshape+transpose trick — 32 dims become 4×8, head axis inserted at dim 1.

Commit before you compute: what does MHA implementation: shapes at every step come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: Output shape: (1, 6, 32) — same as input. Attention weight tensor is (1, 4, 6, 6).

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. MHA is shape-preserving: (B, S, D) in, (B, S, D) out.

Annotate

Walk the callouts on MHA implementation: shapes at every step. Each one is a place this is easy to get subtly wrong.

• The reshape splits the 32-dim into 4 heads × 8 dims each. .transpose(1,2) moves heads before the sequence dim so batched matmul works per head.
• MHA is shape-preserving: (B, S, D) in, (B, S, D) out. The h×S×S attention maps can be inspected per-head to see what each head specialized in.

Discussion prompt

With d_model=32, h=4, d_ff=64: encoder block has 8,416 parameters (verified). FFN typically uses d_ff = 4×d_model, expanding then contracting.

That is stated as though it always holds. Do one of two things: produce a case where it fails, or say precisely what rules such a case out. "It just does" is not on the menu.

Hint: Hunt at the extremes first — zero, one, negative, empty, equal. If every extreme survives, the reason they survive is the proof.

Discussion prompt

Explain Decoder block: two attention sublayers by analogy to something with no Machine Learning in it at all — a queue, a recipe, a map, a bank balance, whatever fits. Then say where your analogy breaks.

Hint: An analogy that never breaks is not an analogy, it is the same idea wearing a hat. Find the seam — that is the part that is actually new.

Predict first

A student writes this, and it looks reasonable:

Wrong: Apply a causal (lower-triangular) mask to the encoder self-attention so early tokens cannot see future ones.

It is wrong. Say what breaks — and say it before you turn the page.

Correct: Feels like 'the encoder processes left-to-right so it should not peek ahead.'

Right: The encoder uses bidirectional (no causal) attention — every position sees every other. Only the decoder's self-attention sublayer uses the causal mask.

Why: Feels like 'the encoder processes left-to-right so it should not peek ahead.'

Step through it

Step through Sinusoidal positional encoding one row at a time. What is driving the change, and what would the row after the last one be?

1. Step 1: pos is 0
2. Step 2: pos is 1
3. Step 3: pos is 2
4. Step 4: pos is 3

Comparison matrix

From Architecture variants: one comparison table: refill the Primary use column from what you know. The rest of the table is as it appeared.

Discussion prompt

A GNN updates each node by aggregating neighbor features, then combining with its own state. Mean aggregation + self-loop on a 3-node graph (edges 0↔1, 1↔2) with 4-dim features:

That is stated as though it always holds. Do one of two things: produce a case where it fails, or say precisely what rules such a case out. "It just does" is not on the menu.

Hint: Hunt at the extremes first — zero, one, negative, empty, equal. If every extreme survives, the reason they survive is the proof.

Step through it

Step through Attention complexity: O(n²d) one row at a time. What is driving the change, and what would the row after the last one be?

1. Step 1: n (seq len) is 64
2. Step 2: n (seq len) is 128
3. Step 3: n (seq len) is 256
4. Step 4: n (seq len) is 512

Discussion prompt

Run The transformer authoring recipe (exam speed) with this step confiscated:

Causal mask: torch.tril(torch.ones(T,T)) — decoder self-attn only; encoder uses no mask

Is it still possible? If it is, say what takes its place and what it costs you. If it is not, say exactly what that step was providing that nothing else does.

Hint: A step you can drop for free was never load-bearing. If you cannot drop it, name the thing that goes wrong the moment it is gone.

Discussion prompt

The transformer authoring recipe (exam speed) works on the cases you have just seen. Push it to the edge: what is the most degenerate input it still handles — empty, zero, one item, everything equal — and what is the first case where it stops being true? Name the case, not just "it breaks".

Hint: Try the smallest legal input, then the largest, then the one where two things collide. Methods are specified at their edges; the middle takes care of itself.

Eliminate the wrong options

With d_k = 64, what is the scale factor applied to QKᵀ before the softmax, and why is it necessary?

3 of these 4 are wrong. Strike them one at a time, and say what rules each one out before you strike the next. The survivor is the answer.

• A. Divide by √64 = 8; prevents dot products from growing O(d_k) large and pushing softmax into vanishing-gradient territory
• B. Divide by 64; the dot product sums d_k terms each O(1), so variance grows as d_k, not √d_k
• C. Multiply by √64 = 8; amplifies the signal-to-noise ratio in high-dimensional spaces
• D. Divide by d_k²; normalizes the full covariance matrix of Q and K

Survives elimination: A

Why: Dot product of two d_k-dim unit random vectors has variance d_k, so std dev is √d_k. Dividing by √d_k restores unit variance, keeping softmax inputs in the responsive region where gradients are non-vanishing. With d_k=64, scale = 1/8 = 0.125.

Check your understanding

With d_k = 64, what is the scale factor applied to QKᵀ before the softmax, and why is it necessary?

• A. Divide by √64 = 8; prevents dot products from growing O(d_k) large and pushing softmax into vanishing-gradient territory (correct)
• B. Divide by 64; the dot product sums d_k terms each O(1), so variance grows as d_k, not √d_k
• C. Multiply by √64 = 8; amplifies the signal-to-noise ratio in high-dimensional spaces
• D. Divide by d_k²; normalizes the full covariance matrix of Q and K

Answer: A

Why: Dot product of two d_k-dim unit random vectors has variance d_k, so std dev is √d_k. Dividing by √d_k restores unit variance, keeping softmax inputs in the responsive region where gradients are non-vanishing. With d_k=64, scale = 1/8 = 0.125.

Why B tempts people

The variance of a sum of d_k unit-variance products is d_k, but std dev = √d_k — dividing by the full d_k over-scales, damping signals too aggressively.

Why C tempts people

Multiplying amplifies the dot products, making softmax outputs more peaked (harder) not more stable — the opposite of what scaling aims for.

Why D tempts people

Dividing by d_k² is not derived from any statistical property of the dot product; it would reduce scores to near zero, destroying the attention signal.

Check your understanding

A model processes a source sequence through N encoder blocks (bidirectional attention), then generates a target token-by-token through N decoder blocks using masked self-attention on the target and cross-attention to the encoder output. Which model does this describe?

• A. T5 (correct)
• B. BERT
• C. GPT-2
• D. ViT

Answer: A

Why: T5 is an encoder-decoder (seq2seq) model: the encoder builds a bidirectional representation of the source; the decoder autoregressively generates the target with masked self-attention (causal) and cross-attention to the encoder output. This is the classic Vaswani 2017 architecture adapted for text-to-text pre-training.

Why B tempts people

BERT is encoder-only — it has no decoder blocks and no cross-attention sublayer. It processes a single masked sequence and predicts masked tokens bidirectionally.

Why C tempts people

GPT-2 is decoder-only — it has no encoder blocks. It processes the full context (source + target concatenated) with a single causal attention stack.

Why D tempts people

ViT is also encoder-only, applied to image patches. It has no decoder blocks or cross-attention, and takes a fixed image as input, not a variable-length source sequence.

Eliminate the wrong options

A 12-layer transformer encoder processes sequences of length n=1024 with d_model=768 and h=12 heads (d_k=64 each). If you double the sequence length to n=2048, by what factor does the cost of the attention computation increase?

3 of these 4 are wrong. Strike them one at a time, and say what rules each one out before you strike the next. The survivor is the answer.

• A. 4× (quadratic in n)
• B. 2× (linear in n)
• C. 8× (cubic in n)
• D. 12× (one factor per layer)

Survives elimination: A

Why: Attention cost is O(n²d). Doubling n multiplies n² by 4, so cost increases 4×. Concretely: at n=1024, QKᵀ is 1024² = 1,048,576 entries per head per layer; at n=2048 it is 2048² = 4,194,304 — exactly 4×. The d factor and h and L factors remain constant.

Check your understanding

A 12-layer transformer encoder processes sequences of length n=1024 with d_model=768 and h=12 heads (d_k=64 each). If you double the sequence length to n=2048, by what factor does the cost of the attention computation increase?

• A. 4× (quadratic in n) (correct)
• B. 2× (linear in n)
• C. 8× (cubic in n)
• D. 12× (one factor per layer)

Answer: A

Why: Attention cost is O(n²d). Doubling n multiplies n² by 4, so cost increases 4×. Concretely: at n=1024, QKᵀ is 1024² = 1,048,576 entries per head per layer; at n=2048 it is 2048² = 4,194,304 — exactly 4×. The d factor and h and L factors remain constant.

Why B tempts people

Linear scaling would hold for FFN layers (O(n·d_ff)) but not attention, which has the extra n from K.T in Q@K.T.

Why C tempts people

O(n³) would arise if we did a naive triple loop over all (query, key, value) triplets, but softmax+weighted-sum is O(n²d), not O(n³).

Why D tempts people

The 12 layers scale cost by 12× overall, but the question asks about the effect of doubling n — layers are a constant multiplier independent of n.

Predict first

The table runs: 0 | 2.3599 | 0.1125 | random init, ~10-class chance · 1 | 2.3490 | 0.1125 | loss falling, weights adjusting · 2 | 2.3391 | 0.1125 | stable decay pattern · 3 | 2.3304 | 0.0625 | acc fluctuates — tiny dataset · 4 | 2.3226 | 0.0500 | loss strictly decreasing

In PixelBERT: CLS classification on digits (full…, given the rows so far: what is the next one — the row where Step is 5?

Correct: 5 | 2.3156 | 0.0875 | longer training needed for high acc

Why: The relationship between the columns, not the individual numbers, is what generates the next row. This exercises the full BERT-style pipeline in < 30 lines: patch projection, CLS prepend, learned positional embedding, encoder block, linear head.

Comparison matrix

From PixelBERT: CLS classification on digits (full…: every row here is a choice with a cost. Fill the Acc column, then say which row you would actually pick and what you give up for it.

Discussion prompt

Your turn — multi-head attention in 15 minutes — before any calculation: what is the plan? Name the moves in order, in plain English, without doing the arithmetic.

Hint: It starts with: Milestone 1 (3 min): Define __init__ — four `nn.Linear(d_model…

Comparison matrix

From Your turn — multi-head attention in 15 minutes: refill the Check column from what you know. The rest of the table is as it appeared.

Draw it

One page, no notation unless you need it: draw how these connect — Scaled Dot-Product Attention — the atomic unit · Multi-Head Attention — h projections in parallel · Encoder & Decoder Blocks — the full architecture · BERT / GPT / T5 / ViT / GNN — the variant landscape · Complexity & Implementation Drill. Put an arrow wherever one of them is what makes another possible, and label the arrow with why.