Lesson 108: Summarization, BART, Perplexity & Calibration

1. Summarization, BART Perplexity & Calibration

Title

USAAIO · Lesson 108 · Phase 3

Abstractive vs extractive summarization, BART denoising pretraining, Pegasus gap-sentence masking, perplexity as exponentiated NLL, and ECE calibration — every number verified in PyTorch.

2. By the end of this lesson you can

Objectives

1. Distinguish extractive from abstractive summarization and state the ROUGE-1/2 asymmetry that reveals the difference
2. Describe BART's four corruption strategies (text infilling, token deletion, sentence permutation, document rotation) and explain why a denoising objective trains a strong encoder-decoder
3. Explain Pegasus gap-sentence masking — how sentence importance is scored, why it aligns pretraining with the summarization objective
4. Compute perplexity by hand from a token-level log-prob table and interpret low/high/uniform-baseline values
5. Explain calibration and ECE — read a reliability diagram, identify an overconfident model, and state when perplexity is a misleading metric

3. Extractive vs Abstractive — what ROUGE reveals

Section

Part 1 of 4

4. Extractive summarization: copy, do not generate

Concept

Extractive summarization selects and concatenates complete sentences (or phrases) from the source document. No new tokens are generated — the summary is a subset of the input.

• Faithful by construction: every word existed in the source — hallucination is impossible
• Rigid: cannot merge, paraphrase, or generalize across sentences
• High ROUGE against the source: n-gram overlap is trivially inflated because tokens are copied

Classic extractive systems: TextRank (graph-based sentence scoring), BertSum (Lesson 90 BERT encoder + sentence-selection head).

5. Break it if you can: Extractive summarization: copy, do not generate

Counterexample

6. Abstractive summarization: generate new text

Concept

Abstractive summarization generates tokens that may never appear in the source. The model must compress, paraphrase, and synthesize — the same task a human writer performs.

• Flexible: can merge several sentences into one compact claim
• Hallucination risk: the decoder can introduce plausible-sounding but factually incorrect content
• Lower ROUGE against source: paraphrases share fewer surface n-grams even when semantically correct

7. Fill in: generates new tokens for Abstractive summarization: generate new text

Comparison

8. Guess the shape of the answer: ROUGE-1 and ROUGE-2 trace — extractive vs…

Estimation

9. ROUGE-1 and ROUGE-2 trace — extractive vs abstractive

Worked example

Reference: the cat sat on the mat near the window (9 tokens). Compare an extractive summary (literal copy of first 6 tokens) and an abstractive paraphrase. Compute ROUGE-1 and ROUGE-2 F1 for each.

def rouge_n_f1(ref, hyp, n):
    def ngrams(s, n):
        toks = s.lower().split()
        return [tuple(toks[i:i+n]) for i in range(len(toks)-n+1)]
    rb = set(ngrams(ref, n))
    hb = ngrams(hyp, n)
    if not hb or not rb:
        return 0.0
    overlap = sum(1 for g in hb if g in rb)
    prec = overlap / len(hb)
    rec  = overlap / len(rb)
    return 2*prec*rec/(prec+rec) if prec+rec else 0.0

ref = 'the cat sat on the mat near the window'
ext = 'the cat sat on the mat'          # extractive copy
abs_ = 'a feline rested beside the window'  # abstractive

for name, hyp in [('extractive', ext), ('abstractive', abs_)]:
    r1 = rouge_n_f1(ref, hyp, 1)
    r2 = rouge_n_f1(ref, hyp, 2)
    print(f'{name}: ROUGE-1={r1:.4f}  ROUGE-2={r2:.4f}')
# extractive: ROUGE-1=0.8000  ROUGE-2=0.7692
# abstractive: ROUGE-1=0.2667  ROUGE-2=0.1538

10. What each one costs: ROUGE-1 and ROUGE-2 trace — extractive vs…

Trade off

11. Something is wrong here: high ROUGE means better summary

Anomaly

12. Trap: high ROUGE means better summary

Trap

13. Break it on purpose: high ROUGE means better summary

Break the constraint

14. BART — denoising encoder-decoder pretraining

Section

Part 2 of 4

15. BART: bidirectional encoder + autoregressive decoder

Concept

BART (Lewis et al., 2020) is a seq2seq denoising autoencoder: the encoder reads a corrupted version of the text and the decoder learns to reconstruct the original — exactly like a denoising autoencoder, but at the token sequence level.

Architecture: standard Transformer encoder (like BERT, Lesson 90 — bidirectional attention, no causal mask) stacked with a standard Transformer decoder (like GPT, Lesson 91 — causal mask, cross-attention to encoder output). Same shape as the original 'Attention is All You Need' (Lesson 88).

16. BART's four corruption strategies

Concept

BART pretraining applies one or more noise functions to the source text. The decoder must reconstruct the original token sequence regardless of how it was corrupted.

1. Text infilling: replace a contiguous span of tokens with a single [MASK] — the decoder must predict the number of missing tokens as well as their identity. Stronger than BERT's independent masking.
2. Token deletion: delete random tokens with no mask placeholder — the decoder must infer which positions are missing
3. Sentence permutation: shuffle the order of sentences in the document; decoder reconstructs the coherent order
4. Document rotation: rotate the document to begin at a random token; decoder identifies and restores the true start

17. By analogy: BART's four corruption strategies

Analogy

18. PEGASUS: sentence-level masking for summarization

Concept

Pegasus (Zhang et al., 2020) replaces entire sentences with [MASK] tokens, selecting the sentences that are most important to the rest of the document — the same sentences a human would include in a summary.

Importance is scored by ROUGE-1 recall between the candidate sentence and the rest of the document. The top-k scoring sentences are masked (Gap Sentence Generation, GSG). The model learns to generate exactly the content humans use in extractive summaries — aligning pretraining directly with the downstream task.

19. What has to be given first: TinyBART — encoder-decoder forward pass

Missing information

20. TinyBART — encoder-decoder forward pass

Worked example

Build a minimal BART-shaped encoder-decoder with vocab_size=20, d_model=32, n_heads=2, 1 encoder layer, 1 decoder layer. Trace shapes through the forward pass.

import torch, torch.nn as nn

class TinyBART(nn.Module):
    def __init__(self, V=20, d=32, h=2, max_len=10):
        super().__init__()
        self.emb = nn.Embedding(V, d)
        self.pos = nn.Embedding(max_len, d)
        enc_l = nn.TransformerEncoderLayer(d, h, dim_feedforward=64,
                                           batch_first=True, dropout=0.0)
        self.enc = nn.TransformerEncoder(enc_l, num_layers=1)
        dec_l = nn.TransformerDecoderLayer(d, h, dim_feedforward=64,
                                            batch_first=True, dropout=0.0)
        self.dec = nn.TransformerDecoder(dec_l, num_layers=1)
        self.head = nn.Linear(d, V)

    def forward(self, src, tgt):
        S, T = src.shape[1], tgt.shape[1]
        enc_out = self.enc(self.emb(src) + self.pos(torch.arange(S)))
        mask = nn.Transformer.generate_square_subsequent_mask(T)
        dec_out = self.dec(self.emb(tgt)+self.pos(torch.arange(T)),
                           enc_out, tgt_mask=mask, tgt_is_causal=True)
        return self.head(dec_out)

torch.manual_seed(42)
model = TinyBART()
print(sum(p.numel() for p in model.parameters()), 'params')  # 22996
src = torch.randint(0, 20, (2, 5))  # B=2, src_len=5
tgt = torch.randint(0, 20, (2, 4))  # B=2, tgt_len=4
out = model(src, tgt)
print(out.shape)                     # torch.Size([2, 4, 20])

21. Fill in: note for TinyBART — encoder-decoder forward pass

Comparison

22. Perplexity — exponentiated cross-entropy

Section

Part 3 of 4

23. Perplexity: the formula and its meaning

Concept

\[ \text{PPL}(x_1,\ldots,x_N) = \exp\!\left(-\frac{1}{N}\sum_{i=1}^{N} \log P(x_i \mid x_{<i})\right) \]

PPL is the geometric mean inverse probability of each token given its context. Intuitively: a model with PPL=100 is as confused as a model that assigns equal probability to 100 choices at every step.

• Lower PPL = better model: the model assigns higher probability to the actual next tokens
• Uniform baseline: PPL = V (vocab size) — random guessing over V tokens
• Perfect predictor: PPL = 1.0 — assigns probability 1 to every correct token
• Practical range: GPT-2 (1.5B) ≈ 18 on WikiText-103; GPT-4-class models reach 3-5

24. Guess the shape of the answer: Perplexity trace on a 3-token toy LM

Estimation

25. Perplexity trace on a 3-token toy LM

Worked example

Vocab: {0:the, 1:cat, 2:sat}. A toy LM defines a 3×3 log-prob table where row=context token, col=next token. Compute PPL for the cat sat (good) vs sat the cat (bad). Predict before calculating.

import torch, math

# Row = context token (0=the,1=cat,2=sat), Col = next token
logits = torch.tensor([
    [0.1, 2.0, 0.3],    # after 'the':  cat very likely
    [0.2, 0.1, 2.5],   # after 'cat':  sat very likely
    [1.5, 0.2, 0.1],   # after 'sat':  the most likely
])
log_p = torch.log_softmax(logits, dim=-1)
print(log_p.numpy().round(4))
# [[-2.1869 -0.2869 -1.9869]
#  [-2.4748 -2.5748 -0.1748]
#  [-0.4181 -1.7181 -1.8181]]

def ppl(seq, log_p):
    lps = [log_p[seq[i], seq[i+1]].item() for i in range(len(seq)-1)]
    return math.exp(-sum(lps)/len(lps))

good = [0, 1, 2]   # the -> cat -> sat
bad  = [2, 0, 1]   # sat -> the -> cat
print(f'PPL(the cat sat) = {ppl(good, log_p):.4f}')   # 1.2596
print(f'PPL(sat the cat) = {ppl(bad,  log_p):.4f}')   # 1.4226
print(f'PPL(uniform V=3) = 3')

26. Work backwards from the answer: Perplexity trace on a 3-token toy LM

Reverse engineer

27. When perplexity is a good (and bad) metric

Concept

• Good for: comparing two models on the same tokenization and test set — e.g., comparing GPT-4 vs GPT-3 on WikiText-103 with the same vocabulary
• Bad for cross-vocabulary comparison: a model with a 50k-token vocabulary will have lower PPL than a character-level model on the same text — the prediction task is easier when tokens are longer
• Bad for downstream correlation: a lower-PPL language model does not always translate to better performance on GLUE or summarization — PPL measures token prediction, not semantic understanding
• Bad for open-ended generation quality: a model can minimize PPL by hedging (assigning moderate probability to many tokens) without producing coherent text

28. Where does each piece belong: Lesson 108: Summarization, BART, Perplexity…

Sorting

29. Something is wrong here: confusing PPL(text) with PPL(token sequence length N)

Anomaly

30. Trap: confusing PPL(text) with PPL(token sequence length N)

Trap

31. Which of these survive contact with Lesson 108: Summarization, BART, Perplexity…?

Two truths and a lie

32. Calibration — does confidence match accuracy?

Section

Part 4 of 4

33. Calibration and Expected Calibration Error (ECE)

Concept

A model is calibrated when its predicted probability matches the empirical frequency of being correct. If the model says 70% on 1,000 examples, roughly 700 should be correct. Modern deep networks are typically overconfident — they output 95% when they are right only 70% of the time.

\[ \text{ECE} = \sum_{b=1}^{B} \frac{|\mathcal{B}_b|}{N} \left| \text{acc}(\mathcal{B}_b) - \text{conf}(\mathcal{B}_b) \right| \]

ECE bins predictions by confidence, then computes the weighted average absolute gap between accuracy and confidence within each bin. ECE = 0 = perfectly calibrated; ECE = 0.1 means confidence is off by 10 percentage points on average.

34. Guess the shape of the answer: ECE computation — calibrated vs overconfident

Estimation

35. ECE computation — calibrated vs overconfident

Worked example

Simulate two models on n=200 examples with 5 confidence bins. Compute ECE for each. Predict which model has lower ECE before running.

import numpy as np

def ece(conf, correct, n_bins=5):
    bins = np.linspace(0, 1, n_bins + 1)
    total = 0.0
    for b in range(n_bins):
        mask = (conf >= bins[b]) & (conf < bins[b+1])
        if mask.sum() == 0:
            continue
        bin_acc  = correct[mask].mean()
        bin_conf = conf[mask].mean()
        total += (mask.sum() / len(conf)) * abs(bin_acc - bin_conf)
    return total

np.random.seed(0)
n = 200
conf_cal  = np.random.uniform(0.5, 1.0, n)
corr_cal  = (np.random.rand(n) < conf_cal).astype(float)

conf_over = np.clip(conf_cal * 1.25, 0, 1.0)   # inflated confidence
corr_over = (np.random.rand(n) < conf_cal*0.75).astype(float)

print(f'ECE calibrated:    {ece(conf_cal,  corr_cal):.4f}')   # 0.0296
print(f'ECE overconfident: {ece(conf_over, corr_over):.4f}')  # 0.1862

36. What each one costs: ECE computation — calibrated vs overconfident

Trade off

37. Without one step: The Lesson 108 recipe

Constraint

38. The Lesson 108 recipe

Pattern

1. Extractive vs abstractive: extractive copies sentences → high ROUGE, no hallucination; abstractive generates new text → lower ROUGE, hallucination risk. Use ROUGE + factual-consistency together.
2. BART pretraining: encoder reads corrupted text; decoder reconstructs original. Four corruptions: text infilling (span→[MASK]), token deletion (no placeholder), sentence permutation, document rotation.
3. Pegasus GSG: mask the k sentences with highest ROUGE-1 recall vs the rest. Aligns pretraining with summarization — the model learns to generate exactly the most important sentences.
4. Perplexity: PPL = exp(-1/N ∑ log P(x_i|x_<i)). Lower = better. Uniform baseline = V. Only compare models with the same vocabulary and test set.
5. Calibration / ECE: bin predictions by confidence; ECE = weighted |acc - conf| across bins. Overconfident models have high ECE. Fix with temperature scaling (divide logits by T>1 before softmax).
6. When PPL misleads: PPL ignores factual accuracy, penalizes paraphrases, and cannot compare models across vocabulary sizes. Always pair with downstream task metrics.

39. Where does it stop working: The Lesson 108 recipe

Edge cases

40. Rule out three: Check yourself — BART corruption

Elimination

41. Check yourself — BART corruption

Check

Think about what makes each BART corruption strategy unique.

42. Answer it before you see the options: Check yourself — perplexity…

Prediction

43. Check yourself — perplexity interpretation

Check

Apply the PPL formula and the uniform baseline.

44. Rule out three: Check yourself — calibration ECE

Elimination

45. Check yourself — calibration ECE

Check

Reason about the reliability diagram before selecting.

46. Your turn: TinyBART copy-task fine-tuning

Section

Project

47. Project brief

Concept

Build and fine-tune TinyBART — a minimal encoder-decoder transformer — on a synthetic copy task. Track PPL as training progresses to see the loss curve a real summarization fine-tune would produce.

Build rules: print tensor shapes after encoder and decoder; verify causal mask shape is (T, T); log both loss and PPL; use torch.manual_seed(42) for reproducibility.

48. Break it if you can: Project brief

Counterexample

49. Milestone 2 — perplexity from logits

Worked example

Your turn: given model output logits of shape (B, T, V) and labels (B, T), compute per-sequence PPL. Predict the PPL at random initialization for V=10.

Hint: use log_softmax(logits, dim=-1), then gather to select log-prob at each label index, then mean(dim=1) for per-sequence NLL, then exp.

import torch, math

def ppl_from_logits(logits, labels):
    # logits: (B, T, V), labels: (B, T)
    log_p = torch.log_softmax(logits, dim=-1)          # (B, T, V)
    lp_label = log_p.gather(2, labels.unsqueeze(-1))   # (B, T, 1)
    lp_label = lp_label.squeeze(-1)                    # (B, T)
    nll = -lp_label.mean(dim=1)                        # (B,)
    return nll.exp()                                   # (B,) PPL per seq

torch.manual_seed(42)
logits = torch.randn(2, 4, 10)   # random-init, V=10
labels = torch.randint(0, 10, (2, 4))
ppl = ppl_from_logits(logits, labels)
print(ppl)  # tensor([24.2912, 19.5234])  (random init ~V=10 range)

50. Which is which, by shape

Discrimination

51. Milestone 3 — copy-task training curve

Worked example

Your turn: train TinyBART for 50 epochs on 32 random sequences (V=10, seq_len=5). Predict: will PPL converge below 2.0? Estimate at how many epochs.

Hint: the copy task is easy (label = src); Adam at lr=1e-3 converges faster than the transformer default 3e-4. Teacher-forcing (feeding ground truth as decoder input) keeps the loss stable.

import torch, torch.nn as nn, math

torch.manual_seed(42)
VOCAB = 10
model = TinyBART(V=VOCAB, d=32, h=2, max_len=8)  # from Milestone 1
opt   = torch.optim.Adam(model.parameters(), lr=1e-3)
loss_fn = nn.CrossEntropyLoss()

src_data = torch.randint(0, VOCAB, (32, 5))
labels   = src_data                      # copy task: predict src

for epoch in range(51):
    opt.zero_grad()
    logits = model(src_data, src_data)   # teacher-force tgt = src
    loss   = loss_fn(logits.reshape(-1, VOCAB), labels.reshape(-1))
    loss.backward(); opt.step()
    if epoch % 10 == 0:
        ppl = math.exp(loss.item())
        print(f'epoch {epoch:3d}  loss={loss.item():.4f}  PPL={ppl:.2f}')
# epoch   0  loss=2.3997  PPL=11.02
# epoch  10  loss=1.7515  PPL=5.76
# epoch  20  loss=1.2176  PPL=3.38
# epoch  30  loss=0.8125  PPL=2.25
# epoch  40  loss=0.5339  PPL=1.71
# epoch  50  loss=0.3535  PPL=1.42

52. The full program

Concept

import torch, torch.nn as nn, math

class TinyBART(nn.Module):
    def __init__(self, V=10, d=32, h=2, max_len=8):
        super().__init__()
        self.emb = nn.Embedding(V, d)
        self.pos = nn.Embedding(max_len, d)
        self.enc = nn.TransformerEncoder(
            nn.TransformerEncoderLayer(d, h, dim_feedforward=64,
                                       batch_first=True, dropout=0.0), 1)
        self.dec = nn.TransformerDecoder(
            nn.TransformerDecoderLayer(d, h, dim_feedforward=64,
                                       batch_first=True, dropout=0.0), 1)
        self.head = nn.Linear(d, V)

    def forward(self, src, tgt):
        S, T = src.shape[1], tgt.shape[1]
        enc_out = self.enc(self.emb(src)+self.pos(torch.arange(S)))
        mask = nn.Transformer.generate_square_subsequent_mask(T)
        dec_out = self.dec(self.emb(tgt)+self.pos(torch.arange(T)),
                           enc_out, tgt_mask=mask, tgt_is_causal=True)
        return self.head(dec_out)

torch.manual_seed(42)
VOCAB = 10
model   = TinyBART(V=VOCAB)
opt     = torch.optim.Adam(model.parameters(), lr=1e-3)
loss_fn = nn.CrossEntropyLoss()
src     = torch.randint(0, VOCAB, (32, 5))
for ep in range(51):
    opt.zero_grad()
    loss = loss_fn(model(src, src).reshape(-1, VOCAB), src.reshape(-1))
    loss.backward(); opt.step()
    if ep % 10 == 0:
        print(f'ep {ep:3d}  PPL={math.exp(loss.item()):.2f}')
# ep   0  PPL=11.02
# ep  10  PPL=5.76
# ep  20  PPL=3.38
# ep  30  PPL=2.25
# ep  40  PPL=1.71
# ep  50  PPL=1.42

53. Fill in: rationale for The full program

Comparison

54. Show it off

Concept

Out loud, slides closed: (1) explain why extractive summarization always outscores abstractive on ROUGE-1 against the source; (2) walk through all four BART corruption strategies and state which one is unique because the decoder must infer span length; (3) compute PPL by hand given two log-prob values.

Homework (from the lesson plan): fine-tune BART-base for abstractive summarization on CNN/DailyMail (use Hugging Face datasets); implement a perplexity function that works for any autoregressive language model; compare extractive (first-3-sentences baseline) vs abstractive (BART) using ROUGE and at least one factual-consistency metric.

55. Connect it up: Lesson 108: Summarization, BART, Perplexity & Calibration

Connect it up

56. What you can do now

Recap

• Explain extractive vs abstractive summarization and why ROUGE favors extractive even when abstractive is semantically better
• Name all four BART corruption strategies and state which requires predicting span length (text infilling)
• Explain Pegasus GSG: mask sentences scored highest by ROUGE-1 recall vs rest of doc
• Compute perplexity from log-prob table: PPL = exp(-1/N ∑ log P(x_i|x_<i)); uniform baseline = V
• Compute ECE from confidence bins and identify overconfidence; apply temperature scaling as the fix