Lesson 84: Transformer Encoder for Classification

Discussion prompt

Before we open Lesson 84: Transformer Encoder for Classification: without looking back, what was the main idea of Positional Encoding, and what could you do by the end of it that you could not do before?

Hint: One sentence for the idea, one for the skill. If the second one is blank, that is the part to revisit.

Discussion prompt

A single encoder block = multi-head self-attention → Add&Norm → position-wise FFN → Add&Norm (Lessons 82–83). Today we stack N of them and add the classification machinery on top.

That is stated as though it always holds. Do one of two things: produce a case where it fails, or say precisely what rules such a case out. "It just does" is not on the menu.

Hint: Hunt at the extremes first — zero, one, negative, empty, equal. If every extreme survives, the reason they survive is the proof.

Comparison matrix

From Parameter count per encoder block: every row here is a choice with a cost. Fill the count (d=64, d_ff=256) column, then say which row you would actually pick and what you give up for it.

Predict first

Build a 2-layer encoder, pass a batch of shape (B=2, T=4, d=16) through it, and confirm the output shape is unchanged.

Commit before you compute: what does Stacking N layers — shape trace come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: output shape = (2, 4, 16), params = 3,264 (= 2 × 1,632 per block)

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. The encoder is shape-preserving: every token embedding is transformed but the (B, T, d) dimensions are identical at input and output — the stack adds depth, not width.

Sort into buckets

Sort these by shape, from memory, without looking back at Stacking N layers — shape trace. Telling them apart on the spot is the skill; the table is only where the answer happens to be written down.

(2, 4, 16)

input x; after layer 1; after layer 2

3,264

params (N=2, d=16, d_ff=32)

g1

shape is "(2, 4, 16)" for input x, after layer 1, after layer 2 — that is what the table on "Stacking N layers — shape trace" records, and it is the single property separating this group from the rest.

g2

shape is "3,264" for params (N=2, d=16, d_ff=32) — that is what the table on "Stacking N layers — shape trace" records, and it is the single property separating this group from the rest.

Comparison matrix

From Sinusoidal PE formula: refill the dim 1 (cos) column from what you know. The rest of the table is as it appeared.

Predict first

Compute the PE matrix for d=8, max_len=10 and print the first 3 rows and 4 columns.

Commit before you compute: what does Building sinusoidal PE in PyTorch come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: PE[:3, :4] = [[0,1,0,1],[0.8415,0.5403,0.0998,0.995],[0.9093,-0.4161,0.1987,0.9801]]

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. Even columns (0, 2) use sin; odd columns (1, 3) use cos.

Step through it

Step through Building sinusoidal PE in PyTorch one row at a time. What is driving the change, and what would the row after the last one be?

1. Step 1: pos is 0
2. Step 2: pos is 1
3. Step 3: pos is 2

Predict first

A student writes this, and it looks reasonable:

The self-attention weights implicitly learn position from training data — no explicit PE needed.

It is wrong. Say what breaks — and say it before you turn the page.

Correct: Without PE, the encoder is a permutation-invariant function: shuffling the tokens produces the same output (just reordered).

Without PE: sorted output_a == sorted output_b — the model cannot distinguish 'A B C' from 'C A B'.

Why: Without PE, the encoder is a permutation-invariant function: shuffling the tokens produces the same output (just reordered). On a dataset with shuffled inputs, accuracy stays at chance level. Experimentally verified: same tokens in different order produce identical sorted output vectors (torch.allclose = True).

Discussion prompt

The rule this trap just fixed:

Without PE: sorted output_a == sorted output_b — the model cannot distinguish 'A B C' from 'C A B'.

Now break it on purpose. Build a case that violates it and follow the consequences until something visibly fails. Where does the failure first show up — and would you have noticed it if you had not been looking?

Hint: The dangerous rules are the ones whose violation still produces an answer. If yours fails loudly, try to find one that fails quietly.

Discussion prompt

Explain The [CLS] token pattern by analogy to something with no Machine Learning in it at all — a queue, a recipe, a map, a bank balance, whatever fits. Then say where your analogy breaks.

Hint: An analogy that never breaks is not an analogy, it is the same idea wearing a hat. Find the seam — that is the part that is actually new.

Discussion prompt

Explain Padding attention mask to a student a year behind you. No notation, no jargon they have not met — and it still has to be true.

Hint: If your explanation needs a symbol they have never seen, you are describing the notation rather than the idea.

Predict first

Run nn.MultiheadAttention on the two-sentence batch with a pad mask and observe that position 3 (PAD) in batch-1 gets zero attention weight.

Commit before you compute: what does MHA with padding mask — attention weights come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: attn_w[1, :, 3] = [0, 0, 0, 0] — the PAD column is zeroed out entirely

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. The key padding mask sets the raw attention score for PAD keys to -inf before softmax, so e^(-inf) = 0.

Step through it

Step through MHA with padding mask — attention weights one row at a time. What is driving the change, and what would the row after the last one be?

1. Step 1: query pos is 0 (CLS)
2. Step 2: query pos is 1 (tok4)
3. Step 3: query pos is 2 (tok5)
4. Step 4: query pos is 3 (PAD)

Discussion prompt

Explain AdamW: decoupled weight decay to a student a year behind you. No notation, no jargon they have not met — and it still has to be true.

Hint: If your explanation needs a symbol they have never seen, you are describing the notation rather than the idea.

Comparison matrix

From Linear warmup + cosine decay schedule: refill the LR (lr_max=1e-3, warmup=10, total=100) column from what you know. The rest of the table is as it appeared.

Discussion prompt

Explain Label smoothing by analogy to something with no Machine Learning in it at all — a queue, a recipe, a map, a bank balance, whatever fits. Then say where your analogy breaks.

Hint: An analogy that never breaks is not an analogy, it is the same idea wearing a hat. Find the seam — that is the part that is actually new.

Predict first

A student writes this, and it looks reasonable:

AdamW(model.parameters(), weight_decay=0.01) — weight decay on all parameters.

It is wrong. Say what breaks — and say it before you turn the page.

Correct: Decaying LayerNorm parameters and biases hurts performance: LayerNorm parameters control the scale of activations, and biases are unconstrained offsets — shrinking them toward 0 is poorly motivated and disrupts training.

Split parameter groups: apply weight_decay only to weight matrices, not biases or norm parameters.

Why: Decaying LayerNorm parameters and biases hurts performance: LayerNorm parameters control the scale of activations, and biases are unconstrained offsets — shrinking them toward 0 is poorly motivated and disrupts training.

Sort into buckets

Some of these hold up and some are the exact mistakes this lesson is built to prevent. Sort them.

Holds up

In PyTorch: nn.TransformerEncoderLayer(d_model, nhead, dim_feedforward, batch_first=True). Wrap N of them with nn.TransformerEncoder(layer, num_layers=N).; For one block with d_model=64, d_ff=256, n_heads=8 — count each sub-module.; Verified for d=8, positions 0–2, dims 0–3 (sin dims only):

Breaks

The self-attention weights implicitly learn position from training data — no explicit PE needed.; AdamW(model.parameters(), weight_decay=0.01) — weight decay on all parameters.

sound

These are stated as this lesson states them — each one survives the edge cases Lesson 84: Transformer Encoder for Classification puts it through.

flawed

Each of these is lifted from a trap in this deck: reasonable-sounding, and wrong in a way that only shows up once you rely on it.

Discussion prompt

Run The transformer encoder classifier recipe with this step confiscated:

Stack N blocks: nn.TransformerEncoder(layer, N) — shape-preserving

Is it still possible? If it is, say what takes its place and what it costs you. If it is not, say exactly what that step was providing that nothing else does.

Hint: A step you can drop for free was never load-bearing. If you cannot drop it, name the thing that goes wrong the moment it is gone.

Discussion prompt

The transformer encoder classifier recipe works on the cases you have just seen. Push it to the edge: what is the most degenerate input it still handles — empty, zero, one item, everything equal — and what is the first case where it stops being true? Name the case, not just "it breaks".

Hint: Try the smallest legal input, then the largest, then the one where two things collide. Methods are specified at their edges; the middle takes care of itself.

Eliminate the wrong options

After N encoder layers, how do we extract a single vector for sequence classification?

3 of these 4 are wrong. Strike them one at a time, and say what rules each one out before you strike the next. The survivor is the answer.

• A. Take encoder_out[:, 0, :] — the [CLS] token at position 0
• B. Average encoder_out over the time dimension (T)
• C. Take encoder_out[:, -1, :] — the last token
• D. Max-pool encoder_out over the time dimension

Survives elimination: A

Why: The [CLS] token is prepended at position 0. Because self-attention is global, after N layers the CLS representation has aggregated context from every other token. We index position 0 with encoder_out[:, 0, :] and pass it to the linear head.

Check your understanding

After N encoder layers, how do we extract a single vector for sequence classification?

• A. Take encoder_out[:, 0, :] — the [CLS] token at position 0 (correct)
• B. Average encoder_out over the time dimension (T)
• C. Take encoder_out[:, -1, :] — the last token
• D. Max-pool encoder_out over the time dimension

Answer: A

Why: The [CLS] token is prepended at position 0. Because self-attention is global, after N layers the CLS representation has aggregated context from every other token. We index position 0 with encoder_out[:, 0, :] and pass it to the linear head.

Why B tempts people

Mean pooling over T is a valid alternative (used in some models) but it mixes in padding-token vectors unless masked; [CLS] pooling is the standard BERT-family approach.

Why C tempts people

In decoder-only models (GPT) the last token aggregates context causally. In the encoder, the last token is arbitrary — there is no causal chain, and padding often sits at the end.

Why D tempts people

Max pooling can work for sentence embeddings (e.g. InferSent) but is not standard in BERT-style encoders and discards positional structure.

Predict first

In nn.MultiheadAttention, the key_padding_mask should be True for positions that are:

Answer it in your own words, now, with nothing to choose from. The options are on the next slide — and picking the right one off a list is an easier skill than producing it.

Correct: Padding tokens — should be ignored

Why: PyTorch's MHA uses True = 'mask out / ignore this position'. So key_padding_mask = (input_ids == pad_id). The masked positions get -inf before softmax, zeroing their attention weight.

Check your understanding

In nn.MultiheadAttention, the key_padding_mask should be True for positions that are:

• A. Padding tokens — should be ignored (correct)
• B. Valid tokens — should attend normally
• C. The [CLS] token — special handling required
• D. The first token of each sequence

Answer: A

Why: PyTorch's MHA uses True = 'mask out / ignore this position'. So key_padding_mask = (input_ids == pad_id). The masked positions get -inf before softmax, zeroing their attention weight.

Why B tempts people

The opposite: True = ignore, False = attend. Passing a mask of valid positions (True=valid) would zero out your actual content and attend only to padding.

Why C tempts people

The [CLS] token is a regular valid token at position 0 — it should not be masked. False for [CLS].

Why D tempts people

Position-based masking is for causal (autoregressive) masks; key_padding_mask is for variable-length sequences, not specific positions.

Eliminate the wrong options

AdamW differs from Adam with L2 regularization because:

3 of these 4 are wrong. Strike them one at a time, and say what rules each one out before you strike the next. The survivor is the answer.

• A. AdamW applies weight decay directly to the parameters, independent of the gradient adaptive scaling
• B. AdamW uses a different moment estimator (β1, β2) than Adam
• C. AdamW decays the learning rate whereas Adam does not
• D. AdamW is Adam with momentum set to 0

Survives elimination: A

Why: Adam+L2 adds λw to the gradient before adaptive scaling, so the effective decay is λ/(sqrt(v̂)+ε) — it varies per parameter and is too small for high-gradient params. AdamW decouples the decay: θ ← θ - η·(grad_update) - ηλθ. The decay is the same for every parameter regardless of its gradient history.

Check your understanding

AdamW differs from Adam with L2 regularization because:

• A. AdamW applies weight decay directly to the parameters, independent of the gradient adaptive scaling (correct)
• B. AdamW uses a different moment estimator (β1, β2) than Adam
• C. AdamW decays the learning rate whereas Adam does not
• D. AdamW is Adam with momentum set to 0

Answer: A

Why: Adam+L2 adds λw to the gradient before adaptive scaling, so the effective decay is λ/(sqrt(v̂)+ε) — it varies per parameter and is too small for high-gradient params. AdamW decouples the decay: θ ← θ - η·(grad_update) - ηλθ. The decay is the same for every parameter regardless of its gradient history.

Why B tempts people

Both Adam and AdamW use the same β1=0.9, β2=0.999 defaults. The moment estimator is unchanged.

Why C tempts people

Neither Adam nor AdamW decays the learning rate by default — that is done by a separate LR scheduler (e.g. LambdaLR). They differ only in how weight decay is applied.

Why D tempts people

Setting β1=0 in Adam gives RMSProp, not AdamW. AdamW keeps the same momentum as Adam.

Discussion prompt

Build a full TransformerEncoderClassifier that embeds tokens, adds sinusoidal PE, applies 2 stacked encoder layers, pools the [CLS] token, and classifies. Train with AdamW + label smoothing.

That is stated as though it always holds. Do one of two things: produce a case where it fails, or say precisely what rules such a case out. "It just does" is not on the menu.

Hint: Hunt at the extremes first — zero, one, negative, empty, equal. If every extreme survives, the reason they survive is the proof.

Step through it

Step through Milestone 1 — sinusoidal PE one row at a time. What is driving the change, and what would the row after the last one be?

1. Step 1: pos is 0
2. Step 2: pos is 1
3. Step 3: pos is 2

Comparison matrix

From Milestone 3 — full model, 50 epochs: every row here is a choice with a cost. Fill the train loss column, then say which row you would actually pick and what you give up for it.

Comparison matrix

From The full program: refill the choice column from what you know. The rest of the table is as it appeared.

Draw it

One page, no notation unless you need it: draw how these connect — Stacking encoder blocks · Positional encoding · [CLS] token & padding mask · AdamW, warmup, label smoothing · Your turn: build the classifier. Put an arrow wherever one of them is what makes another possible, and label the arrow with why.