Lesson 105: BLEU, ROUGE, and BERTScore

1. BLEU · ROUGE & BERTScore

Title

USAAIO · Lesson 105 · Phase 3

Three generations of NLG evaluation: n-gram precision (BLEU), n-gram recall (ROUGE), and contextual embedding similarity (BERTScore). Every formula derived, every number executed.

2. By the end of this lesson you can

Objectives

1. Compute modified n-gram precision and apply the brevity penalty to produce a BLEU-4 score by hand
2. Distinguish BLEU's precision orientation from ROUGE's recall orientation and state why each suits different tasks
3. Implement ROUGE-L using the LCS dynamic-programming table and derive P, R, and F1
4. Explain BERTScore's token-level cosine similarity aggregation and why it captures synonymy that BLEU/ROUGE cannot
5. Identify failure modes of all three metrics and choose the right metric for translation, summarization, and open-ended generation

3. What survived from Text Decoding Strategies?

Warm-up

4. BLEU — Bilingual Evaluation Understudy

Section

Part 1 of 4

5. BLEU: precision of n-gram overlap

Concept

BLEU (Papineni 2002) measures how many n-grams in the hypothesis also appear in at least one reference. It is precision-oriented: it asks 'of what the model produced, how much was also in the reference?'

\[ p_n = \frac{\sum_{\text{hyp}} \min(\text{count}_{\text{hyp}}(g_n),\, \max_r \text{count}_{r}(g_n))}{\sum_{\text{hyp}} \text{count}_{\text{hyp}}(g_n)} \]

The min clip prevents gaming the metric by repeating a common word. BLEU-4 takes the geometric mean of p_1 through p_4, then multiplies by a brevity penalty BP.

\[ \text{BLEU-4} = BP \cdot \exp\!\left(\frac{1}{4}\sum_{n=1}^{4} \log p_n\right) \]

6. Break it if you can: BLEU: precision of n-gram overlap

Counterexample

7. Brevity Penalty: punishing short outputs

Concept

A model that outputs only 'the' has perfect 1-gram precision if 'the' appears in the reference. The brevity penalty BP suppresses this.

\[ BP = \begin{cases} 1 & \text{if } |\text{hyp}| \geq |\text{ref}| \\ e^{1 - |\text{ref}|/|\text{hyp}|} & \text{otherwise} \end{cases} \]

Example: hyp = 'the cat' (2 tokens), ref = 'the cat sat on the mat' (6 tokens). BP = exp(1 - 6/2) = exp(-2) = 0.1353. Even with p_1 = p_2 = 1.0, BLEU is slashed.

8. By analogy: Brevity Penalty: punishing short outputs

Analogy

9. What has to happen first: Worked Example: BLEU-4 from scratch

Ranking

10. Worked Example: BLEU-4 from scratch

Worked example

Hyp: 'it is a guide to action which ensures that the military always obeys the commands of the party' (18 tokens). Ref: 'it is a guide to action that ensures that the military will forever heed party commands' (17 tokens).

from collections import Counter
import math

def get_ngrams(tokens, n):
    return [tuple(tokens[i:i+n]) for i in range(len(tokens)-n+1)]

def modified_precision(hyp, refs, n):
    hyp_ng = Counter(get_ngrams(hyp, n))
    max_ref = Counter()
    for ref in refs:
        ref_ng = Counter(get_ngrams(ref, n))
        for ng in hyp_ng:
            max_ref[ng] = max(max_ref[ng], ref_ng[ng])
    clipped = sum(min(c, max_ref[ng]) for ng, c in hyp_ng.items())
    return clipped, sum(hyp_ng.values())

hyp = "it is a guide to action which ensures that the military always obeys the commands of the party".split()
ref = "it is a guide to action that ensures that the military will forever heed party commands".split()

for n in range(1, 5):
    cn, tn = modified_precision(hyp, [ref], n)
    print(f'p_{n}: {cn}/{tn} = {cn/tn:.4f}')

bp = 1.0  # hyp_len 18 >= ref_len 17
log_sum = sum(math.log(cn/tn) for n in range(1,5)
              for cn, tn in [modified_precision(hyp, [ref], n)])
bleu = bp * math.exp(log_sum / 4)
print(f'BLEU-4 = {bleu:.4f}')

11. Fill in: clipped for Worked Example: BLEU-4 from scratch

Comparison

12. Something is wrong here: treating any p_n = 0 as a recoverable edge case

Anomaly

13. Trap: treating any p_n = 0 as a recoverable edge case

Trap

14. Break it on purpose: treating any p_n = 0 as a recoverable edge…

Break the constraint

15. ROUGE — Recall-Oriented Understudy

Section

Part 2 of 4

16. ROUGE: recall over reference n-grams

Concept

ROUGE (Lin 2004) asks the opposite question from BLEU: 'of the n-grams in the reference, how many did the model produce?' This makes ROUGE suited to summarization, where coverage matters more than precision.

\[ \text{ROUGE-}n = \frac{\sum_{g_n \in \text{ref}} \min(\text{count}_{\text{hyp}}(g_n),\, \text{count}_{\text{ref}}(g_n))}{\sum_{g_n \in \text{ref}} \text{count}_{\text{ref}}(g_n)} \]

ROUGE-1

Unigram recall. Measures word coverage; loose but correlates with fluency.

ROUGE-2

Bigram recall. Captures phrase overlap; stronger signal for content fidelity.

ROUGE-L

Longest Common Subsequence recall. Respects word order without requiring contiguous matches.

17. Which is which: ROUGE: recall over reference n-grams

Matching

18. ROUGE-L: LCS instead of contiguous n-grams

Concept

ROUGE-L uses the longest common subsequence (LCS) — the longest sequence of tokens that appears (in order, not necessarily contiguous) in both hypothesis and reference.

\[ \text{ROUGE-L} = \frac{2 \cdot P_{\text{lcs}} \cdot R_{\text{lcs}}}{P_{\text{lcs}} + R_{\text{lcs}}} \quad P_{\text{lcs}} = \frac{|\text{LCS}|}{|\text{hyp}|} \quad R_{\text{lcs}} = \frac{|\text{LCS}|}{|\text{ref}|} \]

LCS is computed with a standard O(mn) DP table — same algorithm as Edit Distance. A subsequence like ['the','cat','on','the','mat'] skips 'is'/'sat' without breaking.

19. By analogy: ROUGE-L: LCS instead of contiguous n-grams

Analogy

20. What has to happen first: Worked Example: ROUGE-L via LCS DP

Ranking

21. Worked Example: ROUGE-L via LCS DP

Worked example

Hyp: 'the cat sat on the mat' (6 tokens). Ref: 'the cat is on the mat' (6 tokens). Goal: compute ROUGE-L P, R, F1.

def lcs_length(a, b):
    m, n = len(a), len(b)
    dp = [[0]*(n+1) for _ in range(m+1)]
    for i in range(1, m+1):
        for j in range(1, n+1):
            if a[i-1] == b[j-1]:
                dp[i][j] = dp[i-1][j-1] + 1
            else:
                dp[i][j] = max(dp[i-1][j], dp[i][j-1])
    return dp[m][n]

hyp = 'the cat sat on the mat'.split()
ref = 'the cat is on the mat'.split()

lcs = lcs_length(hyp, ref)
p = lcs / len(hyp)
r = lcs / len(ref)
f1 = 2*p*r / (p+r)
print(f'LCS length: {lcs}')
print(f'P={p:.4f}  R={r:.4f}  F1={f1:.4f}')

22. Inspect it line by line: Worked Example: ROUGE-L via LCS DP

Error analysis

23. BERTScore — Contextual Embedding Similarity

Section

Part 3 of 4

24. BERTScore: token-level cosine similarity

Concept

BERTScore (Zhang 2020) encodes hyp and ref through a pretrained BERT model and computes token-level cosine similarity across both sequences, then aggregates with a greedy matching strategy.

\[ P_{\text{BERT}} = \frac{1}{|\hat{x}|} \sum_{\hat{x}_i \in \hat{x}} \max_{x_j \in x} \cos(\mathbf{e}_{\hat{x}_i},\, \mathbf{e}_{x_j}) \]

\[ R_{\text{BERT}} = \frac{1}{|x|} \sum_{x_j \in x} \max_{\hat{x}_i \in \hat{x}} \cos(\mathbf{e}_{x_j},\, \mathbf{e}_{\hat{x}_i}) \]

F1_BERT = harmonic mean of P_BERT and R_BERT. Because contextual embeddings place 'feline' and 'cat' nearby in embedding space, BERTScore correctly assigns high similarity where BLEU-4 and ROUGE-1 score near 0.

25. Break it if you can: BERTScore: token-level cosine similarity

Counterexample

26. Why BERTScore correlates better with humans

Concept

BLEU and ROUGE treat token matching as binary: 'cat' ≠ 'feline' even though they are synonyms. BERTScore uses continuous similarity — any two tokens get a score in [−1, 1] based on their contextual embedding dot product.

BERTScore requires pretrained model weights (typical: bert-base-uncased, roberta-large) and GPU inference — unlike BLEU/ROUGE which are O(N) string ops. This compute cost is the main reason BLEU remains the default MT benchmark.

27. Fill in: BERTScore F1 ≈ for Why BERTScore correlates better with humans

Comparison

28. Something is wrong here: treating ROUGE-1 as sufficient for summarization…

Anomaly

29. Trap: treating ROUGE-1 as sufficient for summarization quality

Trap

30. Which of these survive contact with Lesson 105: BLEU, ROUGE, and BERTScore?

Two truths and a lie

31. Metric Selection and Failure Modes

Section

Part 4 of 4

32. Task-metric alignment

Concept

All three metrics assume at least one reference output exists. For tasks where 'there is no single correct answer' (story generation, chatbots), automatic metrics correlate poorly with human judgment even with BERTScore.

33. What each one costs: Task-metric alignment

Trade off

34. Without one step: Pattern: computing and interpreting NLG metrics

Constraint

35. Pattern: computing and interpreting NLG metrics

Pattern

1. Tokenize hypothesis and reference(s) consistently (same lowercasing, punctuation rules).
2. BLEU-n: for each n ∈ {1..N}, count clipped matches / total hyp n-grams → p_n. If any p_n = 0, BLEU = 0. Apply BP = min(1, exp(1 − |ref| / |hyp|)).
3. BLEU-N = BP × exp((1/N) × Σ log p_n). Typical N = 4 for translation.
4. ROUGE-n: count ref n-gram overlaps / total ref n-grams (recall). Report both n=1 and n=2.
5. ROUGE-L: run O(mn) LCS DP → |LCS|. P = |LCS|/|hyp|, R = |LCS|/|ref|, F1 = harmonic mean.
6. BERTScore: encode both with pretrained BERT; compute pairwise cosine similarity matrix; greedy-match to get P_BERT and R_BERT; take F1.
7. Sanity check: a perfect reference-match should give BLEU=1.0, ROUGE=1.0, BERTScore=1.0. A length-zero hypothesis gives BLEU=0 (BP=0).
8. Choose the right metric for the task — translation → BLEU, summarization → ROUGE, open-ended → BERTScore or human eval.

36. Where does it stop working: Pattern: computing and interpreting NLG…

Edge cases

37. Rule out three: Check 1: BLEU brevity penalty

Elimination

38. Check 1: BLEU brevity penalty

Check

Work through the arithmetic before clicking.

39. Answer it before you see the options: Check 2: ROUGE orientation

Prediction

40. Check 2: ROUGE orientation

Check

Recall the denominator difference between BLEU and ROUGE.

41. Rule out three: Check 3: BERTScore advantage

Elimination

42. Check 3: BERTScore advantage

Check

Think about what BERTScore's embedding similarity captures that BLEU/ROUGE cannot.

43. Your Turn: Implement BLEU and ROUGE-L from scratch

Concept

Project brief: build bleu_rouge.py that (1) implements BLEU-4 with modified n-gram precision and brevity penalty, (2) implements ROUGE-L via LCS DP, and (3) shows a failure-mode example where BLEU=0 but ROUGE-L > 0.

1. Milestone 1: implement count_ngrams(tokens, n) returning a Counter of n-gram tuples.
2. Milestone 2: implement modified_precision(hyp, refs, n) returning (clipped, total). Test: hyp='it is a guide to action which ensures that the military always obeys the commands of the party', ref as in the worked example → p_1=12/18=0.6667.
3. Milestone 3: implement brevity_penalty(hyp_len, ref_len) and bleu4(hyp, refs). Test: same sentence pair → BLEU-4=0.4209.
4. Milestone 4: implement lcs_length(a, b) with O(mn) DP and rouge_l(hyp, ref) returning (P, R, F1). Test: hyp='the cat sat on the mat', ref='the cat is on the mat' → LCS=5, F1=0.8333.
5. Milestone 5: demonstrate a sentence pair where BLEU-4=0 but ROUGE-L F1 > 0 (hint: one of the worked examples above qualifies).

44. Full program: bleu_rouge.py

Worked example

from collections import Counter
import math

# ── N-gram utilities ─────────────────────────────────────────
def count_ngrams(tokens, n):
    return Counter(tuple(tokens[i:i+n])
                   for i in range(len(tokens)-n+1))

def modified_precision(hyp, refs, n):
    hyp_ng = count_ngrams(hyp, n)
    if not hyp_ng:
        return 0, 0
    max_ref = Counter()
    for ref in refs:
        ref_ng = count_ngrams(ref, n)
        for ng in hyp_ng:
            max_ref[ng] = max(max_ref[ng], ref_ng[ng])
    clipped = sum(min(c, max_ref[ng]) for ng, c in hyp_ng.items())
    return clipped, sum(hyp_ng.values())

# ── BLEU-4 ───────────────────────────────────────────────────
def brevity_penalty(hyp_len, ref_len):
    return 1.0 if hyp_len >= ref_len else math.exp(1 - ref_len/hyp_len)

def bleu4(hyp_str, ref_strs):
    hyp = hyp_str.split()
    refs = [r.split() for r in ref_strs]
    ref_len = min((len(r) for r in refs),
                  key=lambda x: abs(x - len(hyp)))
    bp = brevity_penalty(len(hyp), ref_len)
    log_sum = 0.0
    for n in range(1, 5):
        cn, tn = modified_precision(hyp, refs, n)
        if tn == 0 or cn == 0:
            return 0.0
        log_sum += math.log(cn / tn)
    return bp * math.exp(log_sum / 4)

# ── ROUGE-L ──────────────────────────────────────────────────
def lcs_length(a, b):
    m, n = len(a), len(b)
    dp = [[0]*(n+1) for _ in range(m+1)]
    for i in range(1, m+1):
        for j in range(1, n+1):
            if a[i-1] == b[j-1]:
                dp[i][j] = dp[i-1][j-1] + 1
            else:
                dp[i][j] = max(dp[i-1][j], dp[i][j-1])
    return dp[m][n]

def rouge_l(hyp_str, ref_str):
    h, r = hyp_str.split(), ref_str.split()
    lcs = lcs_length(h, r)
    p = lcs / len(h) if h else 0.0
    rc = lcs / len(r) if r else 0.0
    f1 = 2*p*rc/(p+rc) if (p+rc) else 0.0
    return p, rc, f1

# ── Demo ─────────────────────────────────────────────────────
if __name__ == '__main__':
    MT_HYP = ('it is a guide to action which ensures that the '
              'military always obeys the commands of the party')
    MT_REF = ('it is a guide to action that ensures that the '
              'military will forever heed party commands')
    b = bleu4(MT_HYP, [MT_REF])
    print(f'BLEU-4 (MT pair): {b:.4f}')  # 0.4209

    H = 'the cat sat on the mat'
    R = 'the cat is on the mat'
    p, rc, f1 = rouge_l(H, R)
    print(f'ROUGE-L F1:        {f1:.4f}')  # 0.8333
    print(f'BLEU-4 same pair:  {bleu4(H,[R]):.4f}')  # 0.0 (p_4=0)

The third row is the failure-mode: BLEU-4 = 0 because p_4 = 0/3 = 0 (no shared 4-gram). ROUGE-L = 0.8333 because LCS = 5 tokens still capture the shared structure. This motivates reporting both metrics.

45. Fill in: Expected output for Full program: bleu_rouge.py

Comparison

46. Show it off: corner cases to test

Concept

• Repeated-word gaming: hyp = 'the the the the the the', ref = 'the cat sat on the mat'. What does modified_precision return for n=1? Verify the clip prevents inflation.
• Synonym test: hyp = 'a feline rested on the rug', ref = same as above. Show BLEU-4=0, ROUGE-1≈0.33, ROUGE-L F1≈0.33 — and explain why BERTScore would score this higher.
• Multiple references: pass two refs to bleu4. Confirm it picks the best-matching reference length for BP and takes the per-reference max counts for clipping.
• Empty hypothesis: confirm bleu4('', [ref]) returns 0.0 without crashing (BP → 0, and n-gram counts are empty).

47. Connect it up: Lesson 105: BLEU, ROUGE, and BERTScore

Connect it up

48. Lesson 105 recap: BLEU, ROUGE, BERTScore

Recap

• BLEU-4: geometric mean of modified n-gram precisions p_1…p_4, multiplied by BP. Any p_n = 0 collapses the whole score to 0.
• Brevity penalty BP = exp(1 − |ref|/|hyp|) when |hyp| < |ref|, else 1.0. Guards against trivially short outputs.
• ROUGE-n: n-gram recall — denominator is reference length, not hypothesis. ROUGE-2 is the primary content-fidelity signal for summarization.
• ROUGE-L: uses LCS DP (O(mn)) instead of contiguous n-grams; more robust to paraphrase and word reordering.
• BERTScore: pairwise cosine similarity of BERT contextual embeddings; captures synonymy (Lesson 88 self-attention is the backbone); higher human correlation but requires pretrained weights.
• Task alignment: translation → BLEU, summarization → ROUGE-1/2/L, open-ended / semantic tasks → BERTScore or human eval. (See also Lesson 92 on ViT evaluation with accuracy vs. per-class recall.)