Lesson 93: BERT Variants — XLNet, RoBERTa, T5, DeBERTa, ALBERT

1. BERT Variants XLNet · RoBERTa · T5 · DeBERTa · ALBERT

Title

USAAIO · Lesson 93 · Phase 3

Five papers, five distinct fixes to BERT's limitations: inference speed, training signal efficiency, task framing, position attention, and parameter count. Build each core mechanism in PyTorch and understand when to prefer which.

2. By the end of this lesson you can

Objectives

1. State BERT's core inference limitation and trace why bidirectionality causes it
2. Explain XLNet's permutation LM: two-stream attention, content mask vs query mask, and why it avoids the [MASK] token entirely
3. List the three training changes in RoBERTa and explain why dynamic masking captures more signal than static masking
4. Describe the T5 text-to-text framing and write the input format for classification, QA, and summarization
5. Contrast DeBERTa disentangled attention (content+position queries/keys) with standard attention
6. Implement ALBERT cross-layer parameter sharing in PyTorch and verify the parameter reduction factor

3. What survived from Vision Transformer (ViT)?

Warm-up

4. BERT's limitation and why it matters

Section

Part 1 of 5

5. BERT's bidirectionality bottleneck

Concept

BERT (Lesson 90) is bidirectional: the [MASK] training objective lets every token attend to every other token, giving rich contextual representations. But this design has a hidden cost at inference time.

• Encoder-only: BERT cannot generate text autoregressively — it has no causal mask and no decoder
• [MASK] mismatch: training sees [MASK] tokens; inference never does — a distribution shift baked into the architecture
• No left-to-right generation: cannot do translation, summarization, or open-ended generation without bolting on a separate decoder

6. Which is which, by generation

Discrimination

7. XLNet — permutation language model

Section

Part 2 of 5

8. XLNet: predict tokens in a random order

Concept

XLNet trains an autoregressive LM (no [MASK]) but over all permutations of the token order. Predicting token at position t in a permutation means the model has seen all other positions already — capturing bidirectional context without the mask artifact.

\[ \max_\theta \; \mathbb{E}_{\mathbf{z}\sim\mathcal{Z}_T}\left[\sum_{t=1}^{T} \log p_\theta(x_{z_t}\mid x_{\mathbf{z}_{<t}})\right] \]

Here z is a permutation of positions 1..T. The expectation is over all permutations — the model must predict each token from the tokens that appear before it in the permutation order, not the original left-to-right order.

9. Guess the shape of the answer: Two-stream attention — content vs query masks

Estimation

10. Two-stream attention — content vs query masks

Worked example

XLNet needs two attention streams because the model must both use position t's content (for other positions to attend to) and predict it without seeing itself. Standard self-attention cannot do both simultaneously.

import torch

def permutation_masks(seq_len, target_pos, seed=7):
    torch.manual_seed(seed)
    perm = torch.randperm(seq_len)   # e.g. [3,0,1,5,2,4]
    # rank of each position in the permutation
    perm_rank = torch.zeros(seq_len, dtype=torch.long)
    for rank, pos in enumerate(perm):
        perm_rank[pos] = rank
    # content mask: position i can attend j if rank(j) <= rank(i)
    c_mask = (perm_rank.unsqueeze(0) <= perm_rank.unsqueeze(1)).float()
    # query mask: same but target cannot attend to itself
    q_mask = c_mask.clone()
    q_mask[target_pos, target_pos] = 0.0
    return perm, perm_rank, c_mask, q_mask

perm, ranks, cm, qm = permutation_masks(seq_len=6, target_pos=3)
print('permutation:', perm.tolist())
print('perm ranks: ', ranks.tolist())
print('target=3 is rank', ranks[3].item())
print('content mask row 3:', cm[3].int().tolist())
print('query   mask row 3:', qm[3].int().tolist())

11. Fill in: can see past context for Two-stream attention — content vs query masks

Comparison

12. Something is wrong here: XLNet still uses positional encoding for the original…

Anomaly

13. Trap: XLNet still uses positional encoding for the original order

Trap

14. RoBERTa — removing BERT's training shortcuts

Section

Part 3 of 5

15. RoBERTa: three targeted ablations

Concept

RoBERTa (Liu 2019) is BERT with better training — same architecture, three training changes that each demonstrably improve downstream performance.

1. Remove NSP (Next Sentence Prediction): BERT's NSP task was too easy — models solved it with superficial cues. Dropping it forces the MLM to do more work and improves sentence-level tasks.
2. Dynamic masking: generate a fresh mask each epoch instead of a fixed mask at preprocessing time. Each pass sees different positions masked — more training signal per token.
3. Larger batches + more data: 256 → 8,000 sequences per batch; 16 GB → 160 GB text. Bigger batches stabilize the gradient and let the lr be scaled up.

16. Break it if you can: RoBERTa: three targeted ablations

Counterexample

17. What has to be given first: Dynamic masking in code

Missing information

18. Dynamic masking in code

Worked example

Static masking (BERT): mask is computed once at preprocessing; every epoch trains on the same masked positions. Dynamic masking (RoBERTa): each forward pass draws a fresh mask at 15% independently.

import numpy as np

def static_mask(seq_len, mask_prob=0.15, seed=42):
    rng = np.random.RandomState(seed)
    return (rng.random(seq_len) < mask_prob).astype(int)

def dynamic_mask(seq_len, mask_prob=0.15, seed=None):
    rng = np.random.RandomState(seed)
    return (rng.random(seq_len) < mask_prob).astype(int)

seq_len = 20
static = static_mask(seq_len, seed=42)
print('static (same every epoch):  ', static.tolist())
for ep, s in enumerate([42, 99, 7, 13], 1):
    print(f'dynamic epoch {ep} (seed={s}): ', dynamic_mask(seq_len, seed=s).tolist())

19. What each one costs: Dynamic masking in code

Trade off

20. T5 — text-to-text unification

Section

Part 4 of 5

21. T5: every NLP task is seq2seq

Concept

T5 (Raffel 2020) uses a single encoder-decoder for every task. The task type is specified by a text prefix in the input; the model always generates a text string as output. There is no task-specific head architecture.

Even regression (similarity score 0-5) is cast as text generation: the model outputs the digit string "3.8". This unification lets T5 train on all tasks simultaneously with a single cross-entropy loss over the output vocabulary.

22. Guess the shape of the answer: T5-style encoder-decoder forward pass

Estimation

23. T5-style encoder-decoder forward pass

Worked example

Trace a minimal T5 encoder-decoder with vocab=32, d=64, 2 encoder layers, 2 decoder layers. Input: src tokens (batch=2, len=8). Target: tgt tokens (batch=2, len=3). Predict the logit shape.

import torch, torch.nn as nn
torch.manual_seed(42)

VOCAB, D, N_HEADS = 32, 64, 4
enc_layer = nn.TransformerEncoderLayer(D, N_HEADS, dim_feedforward=D*4, batch_first=True)
encoder   = nn.TransformerEncoder(enc_layer, num_layers=2)
dec_layer = nn.TransformerDecoderLayer(D, N_HEADS, dim_feedforward=D*4, batch_first=True)
decoder   = nn.TransformerDecoder(dec_layer, num_layers=2)
embed     = nn.Embedding(VOCAB, D)
out_proj  = nn.Linear(D, VOCAB)

src = torch.randint(0, VOCAB, (2, 8))
tgt = torch.randint(0, VOCAB, (2, 3))
src_e = embed(src)                     # (2, 8, 64)
tgt_e = embed(tgt)                     # (2, 3, 64)
enc_out  = encoder(src_e)              # (2, 8, 64)
# causal mask: decoder can only see left context
causal = nn.Transformer.generate_square_subsequent_mask(3)
dec_out  = decoder(tgt_e, enc_out, tgt_mask=causal)  # (2, 3, 64)
logits   = out_proj(dec_out)           # (2, 3, 32)
print('enc_out:', enc_out.shape)
print('dec_out:', dec_out.shape)
print('logits: ', logits.shape)

24. Work backwards from the answer: T5-style encoder-decoder forward pass

Reverse engineer

25. DeBERTa & ALBERT — attention and size

Section

Part 5 of 5

26. DeBERTa: disentangled content + position attention

Concept

Standard BERT attention fuses content and position into a single query/key vector before computing dot products. DeBERTa disentangles them: it computes three separate score terms and sums them.

\[ A_{i,j} = \underbrace{\mathbf{Q}_i^c \cdot \mathbf{K}_j^c}_{\text{content-content}} + \underbrace{\mathbf{Q}_i^c \cdot \mathbf{K}_{\delta(i,j)}^r}_{\text{content-position}} + \underbrace{\mathbf{Q}_{\delta(i,j)}^r \cdot \mathbf{K}_j^c}_{\text{position-content}} \]

Here delta(i,j) is the relative distance between positions i and j (clipped to a max range). The relative position embedding K^r is shared across all layers. This lets the model independently learn what (content) and where (position) relationships.

27. By analogy: DeBERTa: disentangled content + position attention

Analogy

28. ALBERT: cross-layer parameter sharing

Concept

ALBERT reduces parameter count by sharing the same weight tensor across every encoder layer. Instead of L distinct layers, one layer's weights are applied L times. The model's computational cost (FLOPs) stays the same; only memory drops.

• Sharing attention only, sharing FFN only, or sharing all — ALBERT-base uses all-sharing
• Embedding factorization: vocabulary embedding dim E=128, projected to hidden dim H=768 — reduces embedding params from V·H to V·E + E·H
• Parameter count: ALBERT-base has ~12M params vs BERT-base's ~110M — ~9x smaller; computation stays similar

29. Guess the shape of the answer: ALBERT cross-layer sharing in PyTorch

Estimation

30. ALBERT cross-layer sharing in PyTorch

Worked example

Implement ALBERT-style sharing: one TransformerLayer applied 6 times vs 6 distinct layers. Verify the 6x parameter reduction. Use d=64, n_heads=4.

import torch, torch.nn as nn
torch.manual_seed(42)

class TransformerLayer(nn.Module):
    def __init__(self, d=64, n_heads=4):
        super().__init__()
        self.norm1 = nn.LayerNorm(d)
        self.attn  = nn.MultiheadAttention(d, n_heads, batch_first=True)
        self.norm2 = nn.LayerNorm(d)
        self.mlp   = nn.Sequential(
            nn.Linear(d, d*4), nn.GELU(), nn.Linear(d*4, d)
        )
    def forward(self, x):
        a, _ = self.attn(self.norm1(x), self.norm1(x), self.norm1(x))
        x = x + a
        return x + self.mlp(self.norm2(x))

# Standard 6-layer encoder (6 distinct sets of weights)
class StandardEncoder(nn.Module):
    def __init__(self, n_layers=6):
        super().__init__()
        self.layers = nn.ModuleList([TransformerLayer() for _ in range(n_layers)])
    def forward(self, x):
        for layer in self.layers: x = layer(x)
        return x

# ALBERT-style: one shared layer applied 6 times
class ALBERTEncoder(nn.Module):
    def __init__(self, n_layers=6):
        super().__init__()
        self.shared = TransformerLayer()
        self.n_layers = n_layers
    def forward(self, x):
        for _ in range(self.n_layers): x = self.shared(x)
        return x

std = StandardEncoder()
alb = ALBERTEncoder()
std_p = sum(p.numel() for p in std.parameters())
alb_p = sum(p.numel() for p in alb.parameters())
print(f'Standard 6-layer: {std_p} params')
print(f'ALBERT 6-shared:  {alb_p} params')
print(f'Reduction: {std_p // alb_p}x')
x = torch.randn(2, 10, 64)
print('Output shape:', alb(x).shape)

31. Fill in: output shape for ALBERT cross-layer sharing in PyTorch

Comparison

32. Something is wrong here: ALBERT sharing reduces FLOPs proportional to parameter…

Anomaly

33. Trap: ALBERT sharing reduces FLOPs proportional to parameter count

Trap

34. Break it on purpose: ALBERT sharing reduces FLOPs proportional to…

Break the constraint

35. Without one step: The BERT-variant decision map

Constraint

36. The BERT-variant decision map

Pattern

1. Need generation (summarize/translate)? Use T5 (encoder-decoder, text-to-text) or XLNet (autoregressive perm-LM). BERT alone cannot generate.
2. Fixing BERT's [MASK] artifact? XLNet: permutation LM with two-stream attention (content mask + query mask). No [MASK] token at all.
3. Maximizing BERT's pretraining signal? RoBERTa: drop NSP, use dynamic masking (fresh mask each epoch), scale batches and data.
4. Better position-content interaction? DeBERTa: disentangled Q_c·K_c + Q_c·K_r + Q_r·K_c relative attention.
5. Smaller checkpoint, same depth? ALBERT: shared layer weights + embedding factorization. Saves params/storage; FLOPs unchanged.
6. USAAIO exam heuristic: match the variant to the claimed fix — generation=T5/XLNet, data efficiency=RoBERTa, attention precision=DeBERTa, size=ALBERT.

37. Where does it stop working: The BERT-variant decision map

Edge cases

38. Rule out three: Check yourself — XLNet two-stream

Elimination

39. Check yourself — XLNet two-stream

Check

Work through the logic before clicking.

40. Answer it before you see the options: Check yourself — RoBERTa improvements

Prediction

41. Check yourself — RoBERTa improvements

Check

Identify the exact change.

42. Rule out three: Check yourself — ALBERT vs standard BERT

Elimination

43. Check yourself — ALBERT vs standard BERT

Check

Apply the FLOPs vs params distinction.

44. Your turn: ALBERT sharing + T5 framing

Section

Project

45. Project: implement and compare

Concept

Three milestones: (1) implement ALBERT-style cross-layer sharing and verify the param reduction; (2) wire up a minimal T5 encoder-decoder and trace shapes; (3) write a dynamic masking function and compare masking diversity.

Build rules: use torch.manual_seed(42) for reproducibility; print shapes after every operation; count params with sum(p.numel() for p in model.parameters()) not model.parameters().__len__().

46. Break it if you can: Project: implement and compare

Counterexample

47. Milestone 1 — ALBERT sharing param count

Worked example

Your turn: build StandardEncoder (6 distinct TransformerLayer) and ALBERTEncoder (one shared layer applied 6 times). Predict the ratio before running.

Hint: nn.ModuleList registers multiple distinct layers; assigning one layer to self.shared and calling it N times in forward shares weights. Use sum(p.numel() for p in m.parameters()).

import torch, torch.nn as nn
torch.manual_seed(42)

class TL(nn.Module):
    def __init__(self, d=64, h=4):
        super().__init__()
        self.n1=nn.LayerNorm(d); self.n2=nn.LayerNorm(d)
        self.attn=nn.MultiheadAttention(d, h, batch_first=True)
        self.mlp=nn.Sequential(nn.Linear(d,d*4), nn.GELU(), nn.Linear(d*4,d))
    def forward(self, x):
        a,_=self.attn(self.n1(x),self.n1(x),self.n1(x))
        return x+a+self.mlp(self.n2(x+a))

class Std(nn.Module):
    def __init__(self):
        super().__init__()
        self.layers=nn.ModuleList([TL() for _ in range(6)])
    def forward(self, x):
        for l in self.layers: x=l(x)
        return x

class Alb(nn.Module):
    def __init__(self):
        super().__init__()
        self.shared=TL()
    def forward(self, x):
        for _ in range(6): x=self.shared(x)
        return x

std_p=sum(p.numel() for p in Std().parameters())
alb_p=sum(p.numel() for p in Alb().parameters())
print(f'Standard: {std_p}   ALBERT: {alb_p}   ratio: {std_p//alb_p}x')

48. Milestone 2 — T5 encoder-decoder shapes

Worked example

Your turn: build a minimal encoder-decoder (2 enc layers, 2 dec layers, d=64). Feed src=(2,8) and tgt=(2,3). Trace every tensor shape. Predict the logit shape before running.

Hint: the decoder needs a causal mask (nn.Transformer.generate_square_subsequent_mask(tgt_len)). Cross-attention memory comes from enc_out. The output projection maps from d to vocab_size.

import torch, torch.nn as nn
torch.manual_seed(42)

V, D, H = 32, 64, 4
enc = nn.TransformerEncoder(nn.TransformerEncoderLayer(D,H,D*4,batch_first=True), 2)
dec = nn.TransformerDecoder(nn.TransformerDecoderLayer(D,H,D*4,batch_first=True), 2)
emb = nn.Embedding(V, D)
proj = nn.Linear(D, V)

src = torch.randint(0, V, (2, 8))
tgt = torch.randint(0, V, (2, 3))
enc_out = enc(emb(src))
cm = nn.Transformer.generate_square_subsequent_mask(3)
dec_out = dec(emb(tgt), enc_out, tgt_mask=cm)
logits  = proj(dec_out)
print('enc_out:', enc_out.shape, ' dec_out:', dec_out.shape, ' logits:', logits.shape)

49. What each one costs: Milestone 2 — T5 encoder-decoder shapes

Trade off

50. Milestone 3 — dynamic masking diversity

Worked example

Your turn: write dynamic_mask(seq_len, mask_prob, seed) and call it 4 times with different seeds on a 20-token sequence. Predict whether any two masks will be identical.

Hint: use np.random.RandomState(seed) so each call is reproducible but distinct. Compare static (one mask forever) vs dynamic (new mask each epoch).

import numpy as np

def dynamic_mask(seq_len=20, mask_prob=0.15, seed=None):
    rng = np.random.RandomState(seed)
    masked = np.where(rng.random(seq_len) < mask_prob)[0].tolist()
    return masked

static = dynamic_mask(seed=42)
print('static (fixed):', static)
for ep, s in enumerate([42, 99, 7, 13], 1):
    dm = dynamic_mask(seed=s)
    print(f'epoch {ep} (seed={s}): {dm}')

51. Which is which, by unique?

Discrimination

52. The full program

Concept

import torch, torch.nn as nn, numpy as np
torch.manual_seed(42)

# ---- ALBERT cross-layer sharing ----
class TL(nn.Module):
    def __init__(self, d=64, h=4):
        super().__init__()
        self.n1=nn.LayerNorm(d); self.n2=nn.LayerNorm(d)
        self.attn=nn.MultiheadAttention(d,h,batch_first=True)
        self.mlp=nn.Sequential(nn.Linear(d,d*4),nn.GELU(),nn.Linear(d*4,d))
    def forward(self,x):
        a,_=self.attn(self.n1(x),self.n1(x),self.n1(x))
        return x+a+self.mlp(self.n2(x+a))

class Std(nn.Module):
    def __init__(self): super().__init__(); self.ls=nn.ModuleList([TL() for _ in range(6)])
    def forward(self,x):
        for l in self.ls: x=l(x)
        return x

class Alb(nn.Module):
    def __init__(self): super().__init__(); self.s=TL()
    def forward(self,x):
        for _ in range(6): x=self.s(x)
        return x

std_p=sum(p.numel() for p in Std().parameters())
alb_p=sum(p.numel() for p in Alb().parameters())
print(f'Standard: {std_p}  ALBERT: {alb_p}  ({std_p//alb_p}x reduction)')

# ---- T5 encoder-decoder ----
V,D,H=32,64,4
enc=nn.TransformerEncoder(nn.TransformerEncoderLayer(D,H,D*4,batch_first=True),2)
dec=nn.TransformerDecoder(nn.TransformerDecoderLayer(D,H,D*4,batch_first=True),2)
emb=nn.Embedding(V,D); proj=nn.Linear(D,V)
src=torch.randint(0,V,(2,8)); tgt=torch.randint(0,V,(2,3))
enc_out=enc(emb(src))
cm=nn.Transformer.generate_square_subsequent_mask(3)
logits=proj(dec(emb(tgt),enc_out,tgt_mask=cm))
print('T5 logits:', logits.shape)

# ---- dynamic masking ----
def dyn(n=20,p=0.15,s=None):
    rng=np.random.RandomState(s)
    return np.where(rng.random(n)<p)[0].tolist()

for ep,s in enumerate([42,99,7,13],1):
    print(f'epoch {ep}: {dyn(seed=s)}')

53. Fill in: verified output for The full program

Comparison

54. Show it off

Concept

Out loud, slides closed: (1) explain the XLNet content mask vs query mask distinction and why the target's self-link is zeroed; (2) list RoBERTa's three changes to BERT training; (3) explain why ALBERT's parameter reduction does not translate to faster inference.

Stretch (homework from the lesson plan): load T5 from HuggingFace (from transformers import T5ForConditionalGeneration) and use it for QA as seq2seq. Compare its cross-attention patterns with a BERT-based extractive QA model. Implement ALBERT-style embedding factorization: nn.Embedding(vocab, E=128) followed by nn.Linear(128, 768) and measure the param savings vs nn.Embedding(vocab, 768) for vocab=30,000.

55. Connect it up: Lesson 93: BERT Variants — XLNet, RoBERTa, T5, DeBERTa, ALBERT

Connect it up

56. What you can do now

Recap

• State BERT's inference limitation: encoder-only, no generation, [MASK] train/test mismatch
• Explain XLNet: permutation LM, two-stream attention, content mask allows self-attention, query mask blocks it for the prediction target
• List RoBERTa's three changes: drop NSP, dynamic masking, larger batches + more data
• Write T5 text-to-text input formats for classification, QA, translation, and regression
• Describe DeBERTa's three attention score terms: content-content, content-position, position-content
• Implement ALBERT cross-layer sharing and verify the N-times parameter reduction with FLOPs unchanged