Lesson 89: GPT & Decoder-Only Transformers

1. GPT & Decoder-Only Transformers

Title

USAAIO · Lesson 89 · Phase 3

Causal language modeling, the GPT-2 architecture, scaling laws, emergent abilities, and autoregressive inference — from the masked attention matrix to greedy generation.

2. By the end of this lesson you can

Objectives

1. Distinguish decoder-only GPT from encoder-decoder transformers: no cross-attention, causal mask only
2. Implement causal self-attention with a lower-triangular mask and explain why all positions train in parallel
3. State the GPT-2 architecture family (layers, d_model, heads) and what the 4× FFN expansion does
4. Apply Chinchilla scaling laws to reason about optimal model-size vs. data-size trade-offs
5. Trace the autoregressive generation loop token by token from a learned distribution

3. GPT = decoder-only transformer

Section

Part 1 of 3

4. What makes GPT decoder-only

Concept

A full encoder-decoder (BERT + T5) has two stacks and cross-attention linking them. GPT removes the encoder entirely — only the decoder stack remains, with no cross-attention sublayer.

Removing cross-attention simplifies the block to: LayerNorm → CausalSelfAttention → residual → LayerNorm → FFN → residual. Every GPT layer is identical.

5. Fill in: BERT (encoder) for What makes GPT decoder-only

Comparison

6. Causal language modeling (CLM)

Concept

CLM trains the model to predict the next token given all preceding tokens. The target is simply the input shifted one position to the right — no labels needed beyond the text itself.

\[ \mathcal{L}_{\text{CLM}} = -\frac{1}{T}\sum_{t=1}^{T} \log P(x_t \mid x_1, x_2, \dots, x_{t-1}) \]

Despite predicting left-to-right, all T positions train in parallel during a forward pass: the causal mask blocks each position from attending to future tokens, so the teacher-forced targets for all positions can be computed in a single matrix multiply.

7. Break it if you can: Causal language modeling (CLM)

Counterexample

8. The causal attention mask

Concept

Token at position t may only attend to positions 1 … t. This is enforced by adding -∞ to all future-position logits before the softmax, making those attention weights exactly 0.

import torch
T = 5
causal_mask = torch.tril(torch.ones(T, T))
print(causal_mask.int())

9. What each one costs: The causal attention mask

Trade off

10. Guess the shape of the answer: Causal attention forward pass (T=4, d_k=4)

Estimation

11. Causal attention forward pass (T=4, d_k=4)

Worked example

Trace token t=0's attention through one head: raw scores, masking, softmax, and the output vector. Seed 7, d_k=4.

import torch, torch.nn.functional as F, math
torch.manual_seed(7)
d_k = 4; T = 4
Q = torch.randn(T, d_k)
K = torch.randn(T, d_k)
V = torch.randn(T, d_k)
scores = Q @ K.T / math.sqrt(d_k)
causal = torch.tril(torch.ones(T, T))
scores = scores.masked_fill(causal == 0, float('-inf'))
attn = F.softmax(scores, dim=-1)
out = attn @ V
print('attn[0]:', attn[0].numpy().round(4))
print('out[0]: ', out[0].numpy().round(4))

12. Watch it run: Causal attention forward pass (T=4, d_k=4)

Pattern

13. Something is wrong here: thinking GPT trains tokens sequentially

Anomaly

14. Trap: thinking GPT trains tokens sequentially

Trap

15. Break it on purpose: thinking GPT trains tokens sequentially

Break the constraint

16. GPT-2 architecture & scaling

Section

Part 2 of 3

17. Inside a GPT block

Concept

Each GPT transformer block applies two residual sub-layers in sequence. The pre-norm variant (GPT-2) applies LayerNorm before each sublayer rather than after — empirically more stable at large scale.

1. x = x + CausalSelfAttention(LayerNorm(x)) — residual + pre-norm
2. x = x + FFN(LayerNorm(x)) — residual + pre-norm
3. FFN: Linear(d, 4d) → GELU → Linear(4d, d) — 4× intermediate expansion

Multi-head split: with d_model=256, n_heads=8, each head sees d_head = 256/8 = 32 dimensions. All heads run in parallel, then concatenate and project back to d_model.

18. By analogy: Inside a GPT block

Analogy

19. What has to be given first: TinyGPT from scratch (2L, d=64, vocab=50)

Missing information

20. TinyGPT from scratch (2L, d=64, vocab=50)

Worked example

Build a 2-layer decoder-only GPT. Components: token embedding, learned positional embedding, n_layers GPT blocks, final LayerNorm, linear head to vocab logits.

import torch, torch.nn as nn, torch.nn.functional as F, math

class CausalSelfAttention(nn.Module):
    def __init__(self, d, nh):
        super().__init__()
        self.nh, self.dh = nh, d//nh
        self.qkv  = nn.Linear(d, 3*d)
        self.proj = nn.Linear(d, d)
    def forward(self, x):
        B,T,C = x.shape
        Q,K,V = self.qkv(x).split(C, dim=-1)
        Q = Q.view(B,T,self.nh,self.dh).transpose(1,2)
        K = K.view(B,T,self.nh,self.dh).transpose(1,2)
        V = V.view(B,T,self.nh,self.dh).transpose(1,2)
        s = (Q @ K.transpose(-2,-1)) / math.sqrt(self.dh)
        mask = torch.tril(torch.ones(T,T,device=x.device))
        s = s.masked_fill(mask==0, float('-inf'))
        out = F.softmax(s,-1) @ V
        return self.proj(out.transpose(1,2).contiguous().view(B,T,C))

class GPTBlock(nn.Module):
    def __init__(self, d, nh):
        super().__init__()
        self.ln1=nn.LayerNorm(d); self.attn=CausalSelfAttention(d,nh)
        self.ln2=nn.LayerNorm(d)
        self.ffn=nn.Sequential(nn.Linear(d,4*d),nn.GELU(),nn.Linear(4*d,d))
    def forward(self,x):
        x=x+self.attn(self.ln1(x)); return x+self.ffn(self.ln2(x))

class TinyGPT(nn.Module):
    def __init__(self,V,d,nh,L,T):
        super().__init__()
        self.tok=nn.Embedding(V,d); self.pos=nn.Embedding(T,d)
        self.blocks=nn.Sequential(*[GPTBlock(d,nh) for _ in range(L)])
        self.ln=nn.LayerNorm(d); self.head=nn.Linear(d,V,bias=False)
        self.T=T
    def forward(self,idx):
        B,t=idx.shape
        x=self.tok(idx)+self.pos(torch.arange(t,device=idx.device))
        return self.head(self.ln(self.blocks(x)))

torch.manual_seed(42)
gpt=TinyGPT(50,64,4,2,8)
print(sum(p.numel() for p in gpt.parameters()))
batch=torch.randint(0,50,(2,8))
print(gpt(batch).shape)

21. Work backwards from the answer: TinyGPT from scratch (2L, d=64, vocab=50)

Reverse engineer

22. GPT-2 scaling family

Concept

GPT-2 (2019) explored scale within the decoder-only paradigm. Four model sizes share the same architecture; only depth, width, and head count differ. All use learned positional embeddings and no next-sentence prediction (NSP) objective.

Context window is 1024 tokens for all GPT-2 variants. Each row multiplies d_model by 4 to get the FFN inner dimension — the 4× rule holds exactly.

23. Watch it run: GPT-2 scaling family

Pattern

24. Chinchilla scaling laws

Concept

Hoffman et al. (2022) showed that GPT-era models were under-trained: given a fixed compute budget C ≈ 6·N·D, optimal performance requires scaling data D and model size N proportionally — roughly D ≈ 20·N tokens.

\[ C \approx 6 \cdot N \cdot D \quad\Rightarrow\quad N^{*} = D^{*} \propto \sqrt{C/6} \]

25. Emergent abilities

Concept

Emergent abilities are capabilities that appear sharply at certain model scales, apparently absent in smaller models. Three key examples for USAAIO:

Few-shot prompting

Provide 3 examples in the context window; GPT infers the task and generalizes — no gradient update.

Zero-shot reasoning

Append "Let's think step by step" to unlock chain-of-thought reasoning on arithmetic and logic problems.

In-context learning

The model adapts to new formats and tasks purely by reading the prompt — no parameter update at test time.

These abilities are not explicitly trained — they emerge from the CLM objective at scale. They are fragile (prompt-sensitive) and inconsistent across model versions.

26. Which is which: Emergent abilities

Matching

27. Something is wrong here: treating few-shot prompting as fine-tuning

Anomaly

28. Trap: treating few-shot prompting as fine-tuning

Trap

29. Which of these survive contact with Lesson 89: GPT & Decoder-Only Transformers?

Two truths and a lie

30. Autoregressive inference & the CLM loop

Section

Part 3 of 3

31. Autoregressive generation

Concept

At inference, GPT generates one token at a time. Each new token is appended to the context and the model runs again — a sequential loop, unlike the parallel training pass.

1. Start with a prompt (seed tokens)
2. Forward pass → logits over vocabulary for the last position only
3. Sample (or take argmax) to get the next token
4. Append token to the sequence, repeat from step 2
5. Stop at an <EOS> token or max length

Greedy decoding (argmax) is deterministic and fast but often repetitive. Temperature sampling (logits / T before softmax) and top-k / top-p filtering add diversity.

32. By analogy: Autoregressive generation

Analogy

33. Guess the shape of the answer: Greedy generation with TinyGPT

Estimation

34. Greedy generation with TinyGPT

Worked example

Generate 5 new tokens from seed [5, 12, 3] using the TinyGPT above. At each step, take the argmax over the vocab logits of the last token position.

def generate(model, start, max_new=5):
    tokens = start.clone()
    for _ in range(max_new):
        ctx = tokens[:, -model.T:]          # cap at max_T=8
        logits = model(ctx)                 # (1, t, V)
        nxt = logits[:, -1, :].argmax(-1, keepdim=True)  # greedy
        tokens = torch.cat([tokens, nxt], dim=1)
    return tokens

torch.manual_seed(42)
gpt2 = TinyGPT(50, 64, 4, 2, 8)
start = torch.tensor([[5, 12, 3]])
out = generate(gpt2, start, max_new=5)
print('generated:', out[0].tolist())
print('new tokens:', out[0, 3:].tolist())

35. Fill in: context (last 3 shown) for Greedy generation with TinyGPT

Comparison

36. Guess the shape of the answer: CLM training step on TinyGPT

Estimation

37. CLM training step on TinyGPT

Worked example

Train TinyGPT on a random synthetic token corpus (8 sequences × 8 tokens). Input = tokens[:, 0:7]; target = tokens[:, 1:8]. Same Lesson 40 loop: zero_grad → forward → loss → backward → step.

import torch, torch.nn.functional as F
torch.manual_seed(42)
gpt_tr = TinyGPT(50, 64, 4, 2, 8)
opt = torch.optim.Adam(gpt_tr.parameters(), lr=1e-3)
data = torch.randint(0, 50, (8, 8))
inp = data[:, :-1]; tgt = data[:, 1:]
for step in range(20):
    opt.zero_grad()
    logits = gpt_tr(inp)                    # (8, 7, 50)
    loss = F.cross_entropy(logits.reshape(-1, 50), tgt.reshape(-1))
    loss.backward(); opt.step()
    if step in [0, 4, 9, 19]:
        print(f'step {step+1:>2}: loss={loss.item():.4f}')

38. Work backwards from the answer: CLM training step on TinyGPT

Reverse engineer

39. Without one step: The GPT build & reasoning recipe

Constraint

40. The GPT build & reasoning recipe

Pattern

1. Architecture: decoder-only = tok_emb + pos_emb → L × GPTBlock → LayerNorm → head; no encoder, no cross-attention
2. Block: pre-LayerNorm → CausalSelfAttn (residual) → pre-LayerNorm → FFN 4× (residual); heads split d_model / n_heads
3. Causal mask: tril(ones(T,T)) — position t attends to 0…t; training is parallel (teacher forcing); inference is sequential
4. Scaling (Chinchilla): optimal D ≈ 20·N; more data beats more parameters at fixed compute
5. Emergent abilities: few-shot / chain-of-thought appear at scale — no gradient update, pure context window conditioning

41. Where does it stop working: The GPT build & reasoning recipe

Edge cases

42. Rule out three: Check yourself — causal mask

Elimination

43. Check yourself — causal mask

Check

Work through the mask logic before clicking.

44. Answer it before you see the options: Check yourself — parameter count

Prediction

45. Check yourself — parameter count

Check

Recall the FFN parameter formula.

46. Rule out three: Check yourself — Chinchilla

Elimination

47. Check yourself — Chinchilla

Check

Apply the scaling law.

48. Your turn: build TinyGPT

Section

Project

49. Project: TinyGPT from scratch

Concept

Implement a 2-layer decoder-only GPT (d=64, n_heads=4, vocab=50, max_T=8), train it on a synthetic token corpus with the CLM objective, then generate new tokens autoregressively.

Build rules: causal mask is torch.tril(torch.ones(T,T)); positional embedding indexes torch.arange(T); input/target are data[:,:-1] / data[:,1:]; same Lesson 40 training loop throughout.

50. Break it if you can: Project: TinyGPT from scratch

Counterexample

51. Milestone 1 — CausalSelfAttention, shape check

Worked example

Your turn: implement CausalSelfAttention with d=64, n_heads=4. Pass a random (2, 8, 64) tensor through it. Predict the output shape.

Hint: QKV projection is one nn.Linear(d, 3d); split into Q, K, V; reshape each to (B, n_heads, T, d_head); apply tril mask after scaling by 1/sqrt(d_head).

import torch, torch.nn as nn, torch.nn.functional as F, math
torch.manual_seed(42)

class CausalSelfAttention(nn.Module):
    def __init__(self, d, nh):
        super().__init__()
        self.nh, self.dh = nh, d//nh
        self.qkv  = nn.Linear(d, 3*d)
        self.proj = nn.Linear(d, d)
    def forward(self, x):
        B,T,C = x.shape
        Q,K,V = self.qkv(x).split(C, dim=-1)
        Q = Q.view(B,T,self.nh,self.dh).transpose(1,2)
        K = K.view(B,T,self.nh,self.dh).transpose(1,2)
        V = V.view(B,T,self.nh,self.dh).transpose(1,2)
        s = (Q @ K.transpose(-2,-1)) / math.sqrt(self.dh)
        m = torch.tril(torch.ones(T,T, device=x.device))
        s = s.masked_fill(m==0, float('-inf'))
        out = F.softmax(s,-1) @ V
        return self.proj(out.transpose(1,2).contiguous().view(B,T,C))

attn = CausalSelfAttention(64, 4)
x = torch.randn(2, 8, 64)
print(attn(x).shape)

52. What stays fixed: Milestone 1 — CausalSelfAttention, shape check

Invariant

53. Milestone 2 — full TinyGPT, param count

Worked example

Your turn: assemble TinyGPT with tok_emb + pos_emb + 2×GPTBlock + LayerNorm + head. Count parameters and verify 107,008.

Hint: head uses nn.Linear(d, vocab_size, bias=False); positional embedding indexes torch.arange(T).unsqueeze(0); forward returns self.head(self.ln(self.blocks(x))).

class GPTBlock(nn.Module):
    def __init__(self,d,nh):
        super().__init__()
        self.ln1=nn.LayerNorm(d); self.attn=CausalSelfAttention(d,nh)
        self.ln2=nn.LayerNorm(d)
        self.ffn=nn.Sequential(nn.Linear(d,4*d),nn.GELU(),nn.Linear(4*d,d))
    def forward(self,x):
        x=x+self.attn(self.ln1(x)); return x+self.ffn(self.ln2(x))

class TinyGPT(nn.Module):
    def __init__(self,V,d,nh,L,T):
        super().__init__()
        self.tok=nn.Embedding(V,d); self.pos=nn.Embedding(T,d)
        self.blocks=nn.Sequential(*[GPTBlock(d,nh) for _ in range(L)])
        self.ln=nn.LayerNorm(d); self.head=nn.Linear(d,V,bias=False); self.T=T
    def forward(self,idx):
        B,t=idx.shape
        x=self.tok(idx)+self.pos(torch.arange(t,device=idx.device))
        return self.head(self.ln(self.blocks(x)))

torch.manual_seed(42)
m = TinyGPT(50,64,4,2,8)
print(sum(p.numel() for p in m.parameters()))

54. What each one costs: Milestone 2 — full TinyGPT, param count

Trade off

55. Milestone 3 & 4 — train then generate

Worked example

Your turn: run the CLM training loop for 20 steps and then greedily generate 5 tokens from seed [5, 12, 3]. Predict: will the initial loss be near log(50) ≈ 3.91?

Hint: inp = data[:,:-1]; tgt = data[:,1:]; loss = F.cross_entropy(logits.reshape(-1, V), tgt.reshape(-1)); generation appends argmax(logits[:,-1,:]) each step.

import torch.nn.functional as F
torch.manual_seed(42)
gpt_tr = TinyGPT(50,64,4,2,8)
opt = torch.optim.Adam(gpt_tr.parameters(), lr=1e-3)
data = torch.randint(0,50,(8,8))
inp=data[:,:-1]; tgt=data[:,1:]
for step in range(20):
    opt.zero_grad()
    loss=F.cross_entropy(gpt_tr(inp).reshape(-1,50),tgt.reshape(-1))
    loss.backward(); opt.step()
    if step in [0,4,9,19]: print(f'step {step+1}: {loss.item():.4f}')
# generate
torch.manual_seed(42)
g=TinyGPT(50,64,4,2,8); tokens=torch.tensor([[5,12,3]])
for _ in range(5):
    logits=g(tokens[:,-8:]); tokens=torch.cat([tokens,logits[:,-1:].argmax(-1,keepdim=True).squeeze(0).unsqueeze(0)],dim=1)
print('generated:', tokens[0].tolist())

56. Fill in: loss for Milestone 3 & 4 — train then generate

Comparison

57. Show it off

Concept

Out loud, slides closed: explain (1) why GPT trains all positions in parallel despite generating left-to-right, (2) the two differences between a GPT block and a full encoder-decoder block, and (3) what Chinchilla says to do when you have a fixed compute budget.

Stretch (homework per lesson plan): implement a 6-layer, 256-dim GPT; train on a small text corpus (Shakespeare characters or Python tokens); demonstrate few-shot prompting by prepending 3 examples; walk through the autoregressive inference loop step by step. Next up: Lesson 90 — RLHF, instruction fine-tuning, and aligning GPT with human preferences.

58. Connect it up: Lesson 89: GPT & Decoder-Only Transformers

Connect it up

59. What you can do now

Recap

• Distinguish decoder-only GPT from encoder-decoder: no cross-attention, causal mask only
• Implement causal self-attention (tril mask, parallel training via teacher forcing)
• Trace GPT-2 scaling family (12–48 layers, 768–1600 dim, 4× FFN) and apply Chinchilla D ≈ 20·N
• Explain emergent abilities (few-shot, CoT) as context-window conditioning — no gradient update
• Run the autoregressive generation loop token by token from a trained distribution