Lesson 113: CLIP — Contrastive Language-Image Pretraining

Discussion prompt

Before we open Lesson 113: CLIP — Contrastive Language-Image Pretraining: without looking back, what was the main idea of Instance Segmentation & Contrastive Learning, and what could you do by the end of it that you could not do before?

Hint: One sentence for the idea, one for the skill. If the second one is blank, that is the part to revisit.

Comparison matrix

From The B×B similarity matrix: refill the meaning column from what you know. The rest of the table is as it appeared.

Discussion prompt

Random initialization gives loss ≈ log(B) (uniform over B choices). Perfect alignment gives loss ≈ 0. For B=4 with all-zero similarities: random_loss = log(4) = 1.3863 (verified).

That is stated as though it always holds. Do one of two things: produce a case where it fails, or say precisely what rules such a case out. "It just does" is not on the menu.

Hint: Hunt at the extremes first — zero, one, negative, empty, equal. If every extreme survives, the reason they survive is the proof.

Put in order

Put the moves of Trace InfoNCE: B=3, T=0.1 into the order they have to happen.

1. Build 4-dim normalized embeddings for 3 image-text pairs
2. Compute S = img_emb @ txt_emb.T, then L = S / 0.1
3. Read output: S diag=[0.9911, 0.9849, 0.9965]; sym loss=0.0321

Why: These are the moves of the worked example in the order it makes them, and each one is set up by the one before it. Use hand-constructed vectors where img_i and txt_i are deliberately close; L2-normalize to put all vectors on the unit sphere so dot-product equals cosine similarity.

Comparison matrix

From Trace InfoNCE: B=3, T=0.1: every row here is a choice with a cost. Fill the CE contribution column, then say which row you would actually pick and what you give up for it.

Predict first

A student writes this, and it looks reasonable:

For each positive pair, compute loss = -log(sigmoid(sim_pos)) and add log(1 - sigmoid(sim_neg)) for each negative.

It is wrong. Say what breaks — and say it before you turn the page.

Correct: This looks like contrastive learning but it is actually binary cross-entropy per pair — sigmoid-based, not softmax-based.

For each positive pair, compute softmax over all B negatives in the batch simultaneously — this is F.cross_entropy(S/T, labels).

Why: This looks like contrastive learning but it is actually binary cross-entropy per pair — sigmoid-based, not softmax-based.

Discussion prompt

The rule this trap just fixed:

Each row is a B-class classification: 'which text in this batch matches image i?' In-batch negatives are free — no separate negative mining.

Now break it on purpose. Build a case that violates it and follow the consequences until something visibly fails. Where does the failure first show up — and would you have noticed it if you had not been looking?

Hint: The dangerous rules are the ones whose violation still produces an answer. If yours fails loudly, try to find one that fails quietly.

Predict first

The table runs: Conv2d(1,8,3,pad=1) + ReLU | (B, 8, 8, 8) | input 8×8 image · AdaptiveAvgPool2d(2) | (B, 8, 2, 2) | → flatten → (B, 32) · Linear(32, 16) + L2-norm | (B, 16) | unit-norm image feats · Embedding(20,16).mean + Linear | (B, 16) | unit-norm text feats · img @ txt.T * 14.29 | (B, B) | logits for InfoNCE · F.cross_entropy (symmetric) | scalar | loss=2.6814 (random init)

In TinyCLIP forward pass in PyTorch, given the rows so far: what is the next one — the row where layer / step is Total params?

Correct: Total params | 1200 | 608 img + 592 txt

Why: The relationship between the columns, not the individual numbers, is what generates the next row. Real CLIP uses ViT-L and GPT-style text transformer; TinyCLIP replaces them with a 3-layer CNN and a bag-of-embeddings to verify the loss pipeline in < 1 second.

Comparison matrix

From TinyCLIP forward pass in PyTorch: refill the output shape column from what you know. The rest of the table is as it appeared.

Discussion prompt

Prompt engineering matters: 'a photo of a {c}' outperforms bare '{c}' because internet captions are natural sentences, not bare labels — the distribution shift is smaller.

That is stated as though it always holds. Do one of two things: produce a case where it fails, or say precisely what rules such a case out. "It just does" is not on the menu.

Hint: Hunt at the extremes first — zero, one, negative, empty, equal. If every extreme survives, the reason they survive is the proof.

Predict first

The table runs: a cat | 0.2163 | 0.0027 | no · a dog | 0.6296 | 0.9893 | YES · a car | 0.2923 | 0.0080 | no

In Zero-shot CLIP with TinyCLIP (PyTorch), given the rows so far: what is the next one — the row where class is a bird?

Correct: a bird | -0.3270 | 0.0000 | no

Why: The relationship between the columns, not the individual numbers, is what generates the next row. Using TinyCLIP (random weights) as a drop-in for real CLIP; structure is identical — only scale differs.

Sort into buckets

Sort these by predicted?, from memory, without looking back at Zero-shot CLIP with TinyCLIP (PyTorch). Telling them apart on the spot is the skill; the table is only where the answer happens to be written down.

no

a cat; a car; a bird

YES

a dog

g1

predicted? is "no" for a cat, a car, a bird — that is what the table on "Zero-shot CLIP with TinyCLIP (PyTorch)" records, and it is the single property separating this group from the rest.

g2

predicted? is "YES" for a dog — that is what the table on "Zero-shot CLIP with TinyCLIP (PyTorch)" records, and it is the single property separating this group from the rest.

Predict first

A student writes this, and it looks reasonable:

Compute logits = img_emb @ txt_emb.T where img_emb and txt_emb come directly from the encoder projection head (not normalized).

It is wrong. Say what breaks — and say it before you turn the page.

Correct: The similarity matrix entries now reflect magnitude differences, not directional alignment.

Apply F.normalize(emb, dim=-1) to both img_emb and txt_emb before the dot product. Then S[i,j] is a cosine similarity in [-1, +1].

Why: The similarity matrix entries now reflect magnitude differences, not directional alignment. A high-norm pair dominates regardless of semantic similarity.

Sort into buckets

Some of these hold up and some are the exact mistakes this lesson is built to prevent. Sort them.

Holds up

The diagonal entries are the B matching pairs; the off-diagonal entries are the B(B-1) non-matching pairs. CLIP pushes diagonal high and off-diagonal low.; Prompt engineering matters: 'a photo of a {c}' outperforms bare '{c}' because internet captions are natural sentences, not bare labels — the distribution shift is smaller.; CLIP opened the contrastive vision-language paradigm; subsequent models each extended it along one architectural dimension while keeping the core InfoNCE objective.

Breaks

For each positive pair, compute loss = -log(sigmoid(sim_pos)) and add log(1 - sigmoid(sim_neg)) for each negative.; Compute logits = img_emb @ txt_emb.T where img_emb and txt_emb come directly from the encoder projection head (not normalized).

sound

These are stated as this lesson states them — each one survives the edge cases Lesson 113: CLIP — Contrastive Language-Image Pretraining puts it through.

flawed

Each of these is lifted from a trap in this deck: reasonable-sounding, and wrong in a way that only shows up once you rely on it.

Comparison matrix

From ALIGN, BLIP-2: three dimensions of extension: every row here is a choice with a cost. Fill the zero-shot IMAGENET top-1 column, then say which row you would actually pick and what you give up for it.

Sort into buckets

Sort these by trainable?, from memory, without looking back at CLIP for generation: conditioning diffusion models. Telling them apart on the spot is the skill; the table is only where the answer happens to be written down.

Frozen

Text conditioning; Image conditioning (Img2Img)

Trained

Denoising

Fixed

Schedule

g1

trainable? is "Frozen" for Text conditioning, Image conditioning (Img2Img) — that is what the table on "CLIP for generation: conditioning…" records, and it is the single property separating this group from the rest.

g2

trainable? is "Trained" for Denoising — that is what the table on "CLIP for generation: conditioning…" records, and it is the single property separating this group from the rest.

g3

trainable? is "Fixed" for Schedule — that is what the table on "CLIP for generation: conditioning…" records, and it is the single property separating this group from the rest.

Put in order

These are the steps of CLIP recipe: 6-step blueprint, scrambled. Put them back in order before the next slide shows you.

1. Encode: pass image through image encoder, caption through text encoder; both output (B, D) raw vectors
2. Normalize: F.normalize(emb, dim=-1) on both to get unit-sphere embeddings
3. Similarity matrix: S = img_emb @ txt_emb.T → (B, B) cosine similarities
4. Scale: L = S / tau where tau = log_tau.exp(), log_tau is a learned parameter
5. Symmetric InfoNCE: loss = (CE(L, labels) + CE(L.T, labels)) / 2; labels = torch.arange(B)
6. Zero-shot at inference: build one text embedding per class name, argmax cosine similarity with query image — no fine-tuning

Why: This is the order the recipe itself gives. Recalling the sequence without the slide in front of you is the difference between recognising the method and being able to run it — most of what goes wrong in practice is a step done out of turn.

Discussion prompt

CLIP recipe: 6-step blueprint works on the cases you have just seen. Push it to the edge: what is the most degenerate input it still handles — empty, zero, one item, everything equal — and what is the first case where it stops being true? Name the case, not just "it breaks".

Hint: Try the smallest legal input, then the largest, then the one where two things collide. Methods are specified at their edges; the middle takes care of itself.

Eliminate the wrong options

For a CLIP batch of B=1024 image-text pairs at random initialization (all cosine similarities ≈ 0), approximately what is the expected InfoNCE loss?

3 of these 4 are wrong. Strike them one at a time, and say what rules each one out before you strike the next. The survivor is the answer.

• A. log(1024) ≈ 6.93
• B. log(2) ≈ 0.69 (binary classification)
• C. 0.0 (random init means no loss)
• D. 1.0 (cross-entropy always starts at 1)

Survives elimination: A

Why: Cross-entropy over B uniform logits = log(B). For B=1024, that is log(1024) ≈ 6.93. Verified: with zero similarities and B=4, loss = log(4) = 1.3863 (real execution). This shows why large batches are harder to optimize from scratch.

Check your understanding

For a CLIP batch of B=1024 image-text pairs at random initialization (all cosine similarities ≈ 0), approximately what is the expected InfoNCE loss?

• A. log(1024) ≈ 6.93 (correct)
• B. log(2) ≈ 0.69 (binary classification)
• C. 0.0 (random init means no loss)
• D. 1.0 (cross-entropy always starts at 1)

Answer: A

Why: Cross-entropy over B uniform logits = log(B). For B=1024, that is log(1024) ≈ 6.93. Verified: with zero similarities and B=4, loss = log(4) = 1.3863 (real execution). This shows why large batches are harder to optimize from scratch.

Why B tempts people

InfoNCE is not a binary (positive vs one negative) loss — it is a B-class softmax where every other pair in the batch is a negative. The denominator sums over all B entries, not just 2.

Why C tempts people

Random init gives uniform logits, not zero loss. Uniform softmax (p=1/B for all) gives CE = -log(1/B) = log(B), not 0. Loss of 0 requires the diagonal to dominate perfectly.

Why D tempts people

Cross-entropy is -log(p_correct); for uniform p_correct = 1/B, the result depends on B. It equals 1 only if B ≈ e ≈ 2.72, not in general.

Predict first

With similarity row [0.90, 0.30, 0.20, -0.10], increasing temperature from T=0.07 to T=2.0 causes the softmax probability of the correct class (index 0) to change from ≈0.9998 to ≈0.3277. What is the practical consequence for training?

Answer it in your own words, now, with nothing to choose from. The options are on the next slide — and picking the right one off a list is an easier skill than producing it.

Correct: Gradients become smaller and less discriminative; the model receives weaker signal to separate matching from non-matching pairs

Why: At T=2.0 the softmax is nearly uniform (p_correct=0.3277, entropy=1.3688), meaning the loss gradient is nearly the same regardless of which pair is matched — weak supervision. At T=0.07 (CLIP default), p_correct=0.9998 and the gradient focuses sharply on the few pairs that are wrong. Small T = hard training signal.

Check your understanding

With similarity row [0.90, 0.30, 0.20, -0.10], increasing temperature from T=0.07 to T=2.0 causes the softmax probability of the correct class (index 0) to change from ≈0.9998 to ≈0.3277. What is the practical consequence for training?

• A. Gradients become smaller and less discriminative; the model receives weaker signal to separate matching from non-matching pairs (correct)
• B. The loss decreases, so the model trains faster and reaches convergence sooner
• C. All cosine similarities become equal, so training stops entirely
• D. The model learns to predict harder negatives more accurately

Answer: A

Why: At T=2.0 the softmax is nearly uniform (p_correct=0.3277, entropy=1.3688), meaning the loss gradient is nearly the same regardless of which pair is matched — weak supervision. At T=0.07 (CLIP default), p_correct=0.9998 and the gradient focuses sharply on the few pairs that are wrong. Small T = hard training signal.

Why B tempts people

A nearly-uniform softmax produces higher loss (−log(0.3277) > −log(0.9998)), not lower. The model needs more steps, not fewer, to converge when the temperature is too large.

Why C tempts people

Temperature scales the logits but does not change the underlying cosine similarities, which depend on the encoder weights. Even at very high T the embeddings keep their directions; gradients still exist but are very small.

Why D tempts people

Higher temperature makes all pairs appear equally hard (nearly uniform), which actually obscures hard negatives rather than teaching the model to focus on them. Hard-negative mining requires a low temperature so genuinely confusable pairs produce larger loss.

Eliminate the wrong options

A CLIP model trained on internet image-caption pairs is tested zero-shot on a medical X-ray classification task with classes ['pleural effusion', 'pneumothorax', 'normal']. Which situation most likely LIMITS zero-shot performance?

3 of these 4 are wrong. Strike them one at a time, and say what rules each one out before you strike the next. The survivor is the answer.

• A. Medical X-rays rarely appear with text descriptions in internet crawls, so image and text embeddings are not well-aligned in the medical domain
• B. CLIP's text encoder cannot process words longer than 6 characters
• C. The InfoNCE loss requires at least B=100 examples per class to generalize
• D. Zero-shot requires fine-tuning the temperature parameter on the target dataset

Survives elimination: A

Why: CLIP's zero-shot strength depends on image-text co-occurrence during pretraining. X-ray images paired with clinical terminology are rare in internet text; the shared embedding space is poorly calibrated for this domain. This is the standard domain-shift argument: CLIP excels on natural-image distributions but degrades on out-of-distribution imagery.

Check your understanding

A CLIP model trained on internet image-caption pairs is tested zero-shot on a medical X-ray classification task with classes ['pleural effusion', 'pneumothorax', 'normal']. Which situation most likely LIMITS zero-shot performance?

• A. Medical X-rays rarely appear with text descriptions in internet crawls, so image and text embeddings are not well-aligned in the medical domain (correct)
• B. CLIP's text encoder cannot process words longer than 6 characters
• C. The InfoNCE loss requires at least B=100 examples per class to generalize
• D. Zero-shot requires fine-tuning the temperature parameter on the target dataset

Answer: A

Why: CLIP's zero-shot strength depends on image-text co-occurrence during pretraining. X-ray images paired with clinical terminology are rare in internet text; the shared embedding space is poorly calibrated for this domain. This is the standard domain-shift argument: CLIP excels on natural-image distributions but degrades on out-of-distribution imagery.

Why B tempts people

CLIP's text encoder is a full transformer (Lesson 88/89) with subword tokenization (BPE, Lesson 97) — it handles arbitrarily long medical terms like 'pneumothorax' without issue. Word length is not a CLIP limitation.

Why C tempts people

Zero-shot inference requires zero labeled examples per class — no training at all on the target task. The 'minimum examples' limitation applies to few-shot fine-tuning, not zero-shot CLIP which uses only text prompts.

Why D tempts people

Temperature is fixed at its pretrained value (log_tau learned during CLIP pretraining) at inference time. Zero-shot requires no target-dataset examples, so no parameter can be tuned — temperature included.

Discussion prompt

Your turn: TinyCLIP full training step — before any calculation: what is the plan? Name the moves in order, in plain English, without doing the arithmetic.

Hint: It starts with: Milestone 1 — build encoders and optimizer

Comparison matrix

From Your turn: TinyCLIP full training step: refill the expected loss column from what you know. The rest of the table is as it appeared.

Draw it

One page, no notation unless you need it: draw how these connect — Contrastive objective — matching pairs in a batch · Dual-encoder architecture — image tower + text tower · Zero-shot transfer — classify without a linear head · CLIP extensions — ALIGN, BLIP, BLIP-2. Put an arrow wherever one of them is what makes another possible, and label the arrow with why.