Lesson 98: Word2Vec, GloVe & Static Word Embeddings

Discussion prompt

Before we open Lesson 98: Word2Vec, GloVe & Static Word Embeddings: without looking back, what was the main idea of Subword Tokenization — BPE, WordPiece, SentencePiece, and what could you do by the end of it that you could not do before?

Hint: One sentence for the idea, one for the skill. If the second one is blank, that is the part to revisit.

Discussion prompt

Maximizing this log-likelihood forces the model to build embeddings where similar-context words occupy nearby regions of the embedding space — the geometry encodes semantic relationships.

That is stated as though it always holds. Do one of two things: produce a case where it fails, or say precisely what rules such a case out. "It just does" is not on the menu.

Hint: Hunt at the extremes first — zero, one, negative, empty, equal. If every extreme survives, the reason they survive is the proof.

Comparison matrix

From Two embedding matrices: refill the shape column from what you know. The rest of the table is as it appeared.

Comparison matrix

From Full softmax bottleneck: every row here is a choice with a cost. Fill the dot products per step column, then say which row you would actually pick and what you give up for it.

Discussion prompt

Explain Negative Sampling — binary classifier by analogy to something with no Machine Learning in it at all — a queue, a recipe, a map, a bank balance, whatever fits. Then say where your analogy breaks.

Hint: An analogy that never breaks is not an analogy, it is the same idea wearing a hat. Find the seam — that is the part that is actually new.

Predict first

Trace one NS step: center=king, context=man, negatives=[noble, prince] with D=4, random init seed 99. Predict the sign of the loss before running.

Commit before you compute: what does NS loss — one step traced come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: pos_score=0.0001 neg_scores=[-0.0156, -0.0166] loss=2.0634

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. Near-zero scores from random init → σ(0)=0.5 for all → loss ≈ -log(0.5)×3 ≈ 2.08; verified at 2.0634.

Discussion prompt

Work backwards. The example finished here:

pos_score=0.0001 neg_scores=[-0.0156, -0.0166] loss=2.0634

What was it asked to do, and what must it have been given? Reconstruct the problem from its answer.

Hint: Every quantity in the result had to enter somewhere. Account for each one.

Predict first

A student writes this, and it looks reasonable:

Negative sampling approximates the softmax denominator by sampling k words, so it converges to the same P(c|w) as full softmax, just faster.

It is wrong. Say what breaks — and say it before you turn the page.

Correct: NS does NOT approximate the softmax.

NS is a different objective — binary classification of real vs noise pairs. It is not an approximation of the softmax normalization.

Why: NS does NOT approximate the softmax. It is a completely different objective — a binary cross-entropy for (word, context) classification. The resulting embeddings optimize a different loss function and implicitly factorize the SPPMI matrix, not the log-probability matrix.

Discussion prompt

The rule this trap just fixed:

NS is a different objective — binary classification of real vs noise pairs. It is not an approximation of the softmax normalization.

Now break it on purpose. Build a case that violates it and follow the consequences until something visibly fails. Where does the failure first show up — and would you have noticed it if you had not been looking?

Hint: The dangerous rules are the ones whose violation still produces an answer. If yours fails loudly, try to find one that fails quietly.

Discussion prompt

Explain CBOW architecture to a student a year behind you. No notation, no jargon they have not met — and it still has to be true.

Hint: If your explanation needs a symbol they have never seen, you are describing the notation rather than the idea.

Predict first

Context words [man, woman] (indices 1, 2) predict center. D=4, V=8. Trace the mean-context vector and the logit scores.

Commit before you compute: what does CBOW forward pass come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: ctx_embeds (2,4) → mean → (4,) → logits (8,) → uniform probs before training

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. Random init → near-uniform softmax over V=8 words.

Sort into buckets

Sort these by shape, from memory, without looking back at CBOW forward pass. Telling them apart on the spot is the skill; the table is only where the answer happens to be written down.

(2, 4)

ctx_embeds (man, woman)

(4,)

ctx_mean

(8,)

logits (scores vs all words); probs

g1

shape is "(2, 4)" for ctx_embeds (man, woman) — that is what the table on "CBOW forward pass" records, and it is the single property separating this group from the rest.

g2

shape is "(4,)" for ctx_mean — that is what the table on "CBOW forward pass" records, and it is the single property separating this group from the rest.

g3

shape is "(8,)" for logits (scores vs all words), probs — that is what the table on "CBOW forward pass" records, and it is the single property separating this group from the rest.

Predict first

Train GloVe on the toy co-occurrence matrix (V=8, D=4, Adagrad, lr=0.05, 200 epochs). All 14 non-zero (i,j) pairs contribute to the loss each step.

Commit before you compute: what does GloVe training — loss trace come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: epoch 0: 0.6736 → epoch 50: 0.0140 → epoch 100: 0.0039 → epoch 200: 0.0003

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. GloVe converges smoothly with Adagrad.

Comparison matrix

From GloVe training — loss trace: refill the GloVe loss column from what you know. The rest of the table is as it appeared.

Predict first

A student writes this, and it looks reasonable:

GloVe minimizes the same cross-entropy loss as Skip-gram — it's equivalent to maximizing log P(context|center) over all word pairs.

It is wrong. Say what breaks — and say it before you turn the page.

Correct: GloVe is a weighted least-squares regression objective on log co-occurrence, not a log-likelihood.

GloVe minimizes a weighted MSE: f(X_ij) * (w_i·w_j + b_i + b_j − log X_ij)². It is a matrix-factorization objective, not a probability model.

Why: GloVe is a weighted least-squares regression objective on log co-occurrence, not a log-likelihood. There are no conditional probabilities, no softmax, and no noise samples. The weighting function f(X_ij) distinguishes it further — rare pairs get downweighted, not discarded.

Sort into buckets

Some of these hold up and some are the exact mistakes this lesson is built to prevent. Sort them.

Holds up

After training, the center embeddings U (or the average (U+W)/2) are used as the word vectors. The context matrix W is discarded or averaged in.; The natural P(context | center) requires a full softmax over all V words — typically 50,000 to 1,000,000 vocabulary entries.; Noise distribution P_n(w) ∝ freq(w)^{3/4} upweights rare words slightly. Mikolov et al. found k=5–20 works well; k=2–5 suffices for large corpora.

Breaks

Negative sampling approximates the softmax denominator by sampling k words, so it converges to the same P(c|w) as full softmax, just faster.; GloVe minimizes the same cross-entropy loss as Skip-gram — it's equivalent to maximizing log P(context|center) over all word pairs.

sound

These are stated as this lesson states them — each one survives the edge cases Lesson 98: Word2Vec, GloVe & Static Word Embeddings puts it through.

flawed

Each of these is lifted from a trap in this deck: reasonable-sounding, and wrong in a way that only shows up once you rely on it.

Discussion prompt

Explain PMI and what Skip-gram NS really optimizes to a student a year behind you. No notation, no jargon they have not met — and it still has to be true.

Hint: If your explanation needs a symbol they have never seen, you are describing the notation rather than the idea.

Discussion prompt

Explain Linguistic regularities — the geometry by analogy to something with no Machine Learning in it at all — a queue, a recipe, a map, a bank balance, whatever fits. Then say where your analogy breaks.

Hint: An analogy that never breaks is not an analogy, it is the same idea wearing a hat. Find the seam — that is the part that is actually new.

Discussion prompt

Run The static-embedding recipe with this step confiscated:

GloVe: build X (window co-occurrence), minimize Σ f(X_ij)(w_i·w_j + b_i + b_j − log X_ij)² with Adagrad

Is it still possible? If it is, say what takes its place and what it costs you. If it is not, say exactly what that step was providing that nothing else does.

Hint: A step you can drop for free was never load-bearing. If you cannot drop it, name the thing that goes wrong the moment it is gone.

Discussion prompt

The static-embedding recipe works on the cases you have just seen. Push it to the edge: what is the most degenerate input it still handles — empty, zero, one item, everything equal — and what is the first case where it stops being true? Name the case, not just "it breaks".

Hint: Try the smallest legal input, then the largest, then the one where two things collide. Methods are specified at their edges; the middle takes care of itself.

Eliminate the wrong options

In Skip-gram with negative sampling, what does the model learn to do for a positive pair (center w, context c)?

3 of these 4 are wrong. Strike them one at a time, and say what rules each one out before you strike the next. The survivor is the answer.

• A. Maximize σ(v_c · u_w) — push the dot product high so the binary classifier outputs probability ≈ 1
• B. Minimize the full softmax log-probability log P(c|w) summed over the whole vocabulary
• C. Maximize the cosine similarity between u_w and u_c (both center embeddings)
• D. Minimize the L2 distance between u_w and v_c

Survives elimination: A

Why: NS converts the problem to binary classification: push σ(v_c·u_w) toward 1 for real pairs and σ(v_n·u_w) toward 0 for noise pairs. The loss is -log σ(pos) − Σ log σ(−neg). No softmax normalization over V occurs — that is the entire point.

Check your understanding

In Skip-gram with negative sampling, what does the model learn to do for a positive pair (center w, context c)?

• A. Maximize σ(v_c · u_w) — push the dot product high so the binary classifier outputs probability ≈ 1 (correct)
• B. Minimize the full softmax log-probability log P(c|w) summed over the whole vocabulary
• C. Maximize the cosine similarity between u_w and u_c (both center embeddings)
• D. Minimize the L2 distance between u_w and v_c

Answer: A

Why: NS converts the problem to binary classification: push σ(v_c·u_w) toward 1 for real pairs and σ(v_n·u_w) toward 0 for noise pairs. The loss is -log σ(pos) − Σ log σ(−neg). No softmax normalization over V occurs — that is the entire point.

Why B tempts people

That is the full-softmax objective. NS specifically avoids the V-term denominator by replacing it with k noise samples and a binary BCE loss.

Why C tempts people

NS uses the asymmetric dot product v_c·u_w between context and center matrices, not cosine similarity between two center vectors. Using the same embedding matrix for both would conflate the two statistical roles.

Why D tempts people

GloVe minimizes a squared (log-count) error; NS uses log-sigmoid (BCE), not L2 distance in embedding space.

Predict first

You are training word embeddings on a corpus with a long-tail vocabulary (many rare technical terms). Which model/configuration should you prefer?

Answer it in your own words, now, with nothing to choose from. The options are on the next slide — and picking the right one off a list is an easier skill than producing it.

Correct: Skip-gram with negative sampling

Why: Skip-gram generates one (center, context) training pair per co-occurrence, giving rare words proportionally more gradient updates — each occurrence of a rare word contributes 2c independent predictions. CBOW averages context embeddings, washing out the signal for rare words that appear only a few times.

Check your understanding

You are training word embeddings on a corpus with a long-tail vocabulary (many rare technical terms). Which model/configuration should you prefer?

• A. Skip-gram with negative sampling (correct)
• B. CBOW with large window size
• C. GloVe with x_max = 1 (no weight cap)
• D. CBOW with window size 1

Answer: A

Why: Skip-gram generates one (center, context) training pair per co-occurrence, giving rare words proportionally more gradient updates — each occurrence of a rare word contributes 2c independent predictions. CBOW averages context embeddings, washing out the signal for rare words that appear only a few times.

Why B tempts people

CBOW averages all context vectors, which dilutes the signal for rare center words regardless of window size. A larger window makes this averaging worse for rare terms, not better.

Why C tempts people

Setting x_max=1 means f(X_ij) = min(1,(X/1)^0.75) ≥ 1 for all X≥1, removing the soft cap — high-frequency pairs dominate and rare pairs are relatively underweighted, the opposite of what you want.

Why D tempts people

CBOW with window=1 reduces the context to just one neighbor per side, giving fewer training signals per word and hurting both common and rare words.

Eliminate the wrong options

In the GloVe objective J = Σ f(X_ij)(w_i·w_j + b_i + b_j − log X_ij)², what is the purpose of f(X_ij) = min(1, (X_ij/x_max)^α)?

3 of these 4 are wrong. Strike them one at a time, and say what rules each one out before you strike the next. The survivor is the answer.

• A. Downweight very rare pairs (low X_ij) and cap the weight of very frequent pairs (X_ij ≥ x_max) so neither extreme dominates
• B. Convert X_ij into a probability distribution so the objective becomes equivalent to cross-entropy
• C. Apply L2 regularization to the word vectors w_i
• D. Replace log X_ij with a smooth approximation that is defined even when X_ij = 0

Survives elimination: A

Why: f(X_ij) is a soft weight in [0,1]. For small X_ij it is small (rare co-occurrences get low weight, reducing noise). For X_ij ≥ x_max it is capped at 1 (common words like 'the' are not over-represented). The weighting is neither normalization nor regularization — it is an explicit importance weight on the squared residual.

Check your understanding

In the GloVe objective J = Σ f(X_ij)(w_i·w_j + b_i + b_j − log X_ij)², what is the purpose of f(X_ij) = min(1, (X_ij/x_max)^α)?

• A. Downweight very rare pairs (low X_ij) and cap the weight of very frequent pairs (X_ij ≥ x_max) so neither extreme dominates (correct)
• B. Convert X_ij into a probability distribution so the objective becomes equivalent to cross-entropy
• C. Apply L2 regularization to the word vectors w_i
• D. Replace log X_ij with a smooth approximation that is defined even when X_ij = 0

Answer: A

Why: f(X_ij) is a soft weight in [0,1]. For small X_ij it is small (rare co-occurrences get low weight, reducing noise). For X_ij ≥ x_max it is capped at 1 (common words like 'the' are not over-represented). The weighting is neither normalization nor regularization — it is an explicit importance weight on the squared residual.

Why B tempts people

Dividing by f(X_ij) would not produce a valid probability. The objective remains a weighted squared loss — there is no softmax normalization anywhere in GloVe.

Why C tempts people

L2 regularization adds λ‖w‖² to the loss; f(X_ij) is a per-pair importance weight that multiplies the squared residual. These are structurally different terms.

Why D tempts people

GloVe only sums over pairs where X_ij > 0, so log X_ij is always defined. The function f handles importance weighting, not numerical stability at zero.

Discussion prompt

Build Skip-gram NS from scratch — two embedding matrices, a binary loss, a training loop — and watch the embeddings shift so that king-man+woman moves closer to queen.

That is stated as though it always holds. Do one of two things: produce a case where it fails, or say precisely what rules such a case out. "It just does" is not on the menu.

Hint: Hunt at the extremes first — zero, one, negative, empty, equal. If every extreme survives, the reason they survive is the proof.

Sort into buckets

Sort these by shape, from memory, without looking back at Milestone 1 — NS loss one step. Telling them apart on the spot is the skill; the table is only where the answer happens to be written down.

(4,)

u_c (king center embed); v_o (man context embed)

(3, 4)

v_n (3 neg context embeds)

scalar

pos_score u_c·v_o; NS loss

g1

shape is "(4,)" for u_c (king center embed), v_o (man context embed) — that is what the table on "Milestone 1 — NS loss one step" records, and it is the single property separating this group from the rest.

g2

shape is "(3, 4)" for v_n (3 neg context embeds) — that is what the table on "Milestone 1 — NS loss one step" records, and it is the single property separating this group from the rest.

g3

shape is "scalar" for pos_score u_c·v_o, NS loss — that is what the table on "Milestone 1 — NS loss one step" records, and it is the single property separating this group from the rest.

Step through it

Step through Milestone 2 — training loop one row at a time. What is driving the change, and what would the row after the last one be?

1. Step 1: epoch is 0
2. Step 2: epoch is 10
3. Step 3: epoch is 50
4. Step 4: epoch is 100
5. Step 5: epoch is 200

Comparison matrix

From Milestone 3 — analogy evaluation: every row here is a choice with a cost. Fill the note column, then say which row you would actually pick and what you give up for it.

Comparison matrix

From The full program: refill the value column from what you know. The rest of the table is as it appeared.

Draw it

One page, no notation unless you need it: draw how these connect — Skip-gram — predict context from center · CBOW — predict center from context · GloVe — factorize co-occurrences · PMI geometry — why analogies work · Your turn: Word2Vec from scratch. Put an arrow wherever one of them is what makes another possible, and label the arrow with why.