Lesson 29: Mutual Information & Contrastive Learning

1. Mutual Information & Contrastive Learning

Title

USAAIO · Lesson 29 · Week 10 (Information Theory)

How much does knowing one variable tell you about another? We derive I(X;Y) three ways — entropy, chain rule, and KL — skipping no step, compute each by hand on one running joint, then ride it into feature selection and the InfoNCE loss behind CLIP.

2. By the end of this lesson you can

Objectives

1. Recall entropy H(X) = −Σ p log₂ p and read it as bits of uncertainty
2. Derive mutual information three equivalent ways — H(X)+H(Y)−H(X,Y), H(Y)−H(Y|X), and D_KL(p(x,y)‖p(x)p(y)) — and compute all three by hand to the same number
3. Prove I ≥ 0, I ≤ min(H(X),H(Y)), and I = 0 ⟺ X ⊥ Y, then show a zero-correlation pair with large MI
4. Select features by mutual information with the label and read sklearn's scores
5. Trace the InfoNCE contrastive loss on a real batch and connect entropy to compression (MDL)

3. What survived from Kernel Methods?

Warm-up

4. Recap: entropy & one running joint

Section

Part 1 of 7 — the raw material

5. Entropy is uncertainty measured in bits

Intuition

Before mutual information there is entropy — how uncertain you are about a random variable, in bits. A fair coin (50/50) has 1 bit: you need exactly one yes/no answer to pin it down.

A rigged coin that lands heads 99% of the time is almost certain, so it carries less than a bit. A variable you already know carries 0 bits — nothing left to learn.

Mutual information will be entropy that two variables share — how much the uncertainty in one drops once you see the other. So we start by nailing entropy down cold.

6. Break it if you can: Entropy is uncertainty measured in bits

Counterexample

7. Self-information: the surprise of one outcome

Concept

The surprise of a single outcome with probability p is −log₂ p bits. A coin flip (p = 0.5) is −log₂ 0.5 = 1 bit of surprise; a 1-in-8 event is −log₂(1/8) = 3 bits — three times as surprising.

\[ \text{surprise}(x) = -\log_2 p(x) \]

Entropy will just be the average surprise over all outcomes. Rare things surprise more, certain things (p = 1) surprise not at all — −log₂ 1 = 0. Keep this: MI counts how much one variable's surprise drops once you see the other.

8. The entropy formula

Concept

For a discrete variable X with outcome probabilities p(x), entropy is the expected surprise −log₂ p(x), averaged over outcomes:

\[ H(X) = -\sum_{x} p(x)\,\log_2 p(x) \]

Rare outcomes (small p) are surprising (large −log₂ p); certain outcomes (p = 1) carry zero surprise. Using log₂ makes the unit bits. We adopt the convention 0·log 0 = 0.

9. By analogy: The entropy formula

Analogy

10. Guess the shape of the answer: Entropy in code — three sanity checks

Estimation

11. Entropy in code — three sanity checks

Worked example

We mask out zero probabilities (log 0 is undefined but 0·log 0 = 0), then sum. Run this standalone:

import numpy as np

def entropy(p):
    p = np.asarray(p, dtype=float)
    p = p[p > 0]                      # 0 log 0 = 0, so drop zeros
    return -np.sum(p * np.log2(p))    # bits

print(entropy([0.5, 0.5]))            # 1.0
print(entropy([0.4, 0.1, 0.1, 0.4])) # 1.7219...
print(entropy([1.0]))                 # 0.0 (no uncertainty)

12. Fill in: H (bits) for Entropy in code — three sanity checks

Comparison

13. What has to be given first: Entropy peaks at the fair coin

Missing information

14. Entropy peaks at the fair coin

Worked example

Sweep a coin's head-probability q from fair to certain and watch entropy fall. Uniform is maximally uncertain; a rigged coin is more predictable, so it carries fewer bits:

import numpy as np

def entropy(p):
    p = np.asarray(p, dtype=float); p = p[p > 0]
    return -np.sum(p * np.log2(p))

for q in [0.5, 0.75, 0.9, 0.99, 1.0]:
    print(q, round(entropy([q, 1-q]), 4))

15. What each one costs: Entropy peaks at the fair coin

Trade off

16. The running example: weather × umbrella

Concept

One dataset carries the whole lesson. X is the weather (0 = dry, 1 = rain); Y is whether a commuter carries an umbrella (0 = no, 1 = yes). Their joint distribution p(x,y):

The mass sits on the diagonal: dry-and-no-umbrella and rain-and-umbrella are common; the off-diagonal mismatches are rare. So the two are related — but not perfectly. Quantifying exactly how related is the job of mutual information.

17. Teach it back: The running example: weather × umbrella

Explain it

18. Predict the next row: Marginals: collapse the joint

Pattern

19. Marginals: collapse the joint

Worked example

The marginal p(x) sums each row (over all y); p(y) sums each column (over all x). Sum over the axis you want to erase:

import numpy as np

J = np.array([[0.4, 0.1],
              [0.1, 0.4]])           # rows = X, cols = Y
px = J.sum(axis=1)                    # marginal of X: sum over Y
py = J.sum(axis=0)                    # marginal of Y: sum over X
print("px =", px)                     # [0.5 0.5]
print("py =", py)                     # [0.5 0.5]
print("sum J =", J.sum())             # 1.0

20. Work backwards from the answer: Marginals: collapse the joint

Reverse engineer

21. Complete the line: Compute H(X), H(Y), H(X,Y) by hand

Fill the middle

22. Compute H(X), H(Y), H(X,Y) by hand

Worked example

Three entropies feed every MI formula. Marginals are [0.5, 0.5], so both give 1 bit:

\[ H(X) = -2\big(0.5\log_2 0.5\big) = -2(0.5)(-1) = 1 \text{ bit} \]

The joint entropy uses all four cells [0.4, 0.1, 0.1, 0.4]:

\[ H(X,Y) = -2\big(0.4\log_2 0.4\big) - 2\big(0.1\log_2 0.1\big) \]

23. Why H(X,Y) < H(X) + H(Y)

Concept

Notice H(X) + H(Y) = 2.0 but H(X,Y) = 1.7219. The joint is less uncertain than the two separately. If X and Y were independent they'd add to exactly 2.0; the shortfall is real structure.

That gap — 2.0 − 1.7219 = 0.2781 bits — is precisely the information they share. Hold that number: it is the mutual information, and the next part earns it three different ways.

24. Mutual information: three faces

Section

Part 2 of 7 — one quantity, three derivations

25. Picture it first: MI is shared uncertainty

Picture it

26. MI is shared uncertainty

Intuition

Picture two overlapping circles: H(X) is the left circle, H(Y) the right. Where they overlap is information the two variables have in common — see one, and you learn something about the other.

That overlap is mutual information I(X;Y). If the circles are disjoint (no overlap) the variables are independent and I = 0. If one circle sits entirely inside the other, knowing one fully determines the shared part.

27. The three equivalent forms

Concept

Mutual information is one number wearing three algebraic costumes. Each will be derived in full — but here they are up front so you know the destination:

\[ I(X;Y) = H(X) + H(Y) - H(X,Y) \]

\[ \;\;= H(Y) - H(Y\mid X) \]

\[ \;\;= D_{KL}\big(p(x,y)\,\|\,p(x)p(y)\big) \]

Form 1 is the set overlap. Form 2 is uncertainty reduced by observing X. Form 3 is the distance from independence. They are equal — provably, not coincidentally.

28. The definition we build from

Concept

The ground-truth definition of MI is the expected pointwise log-ratio between the joint and the product of marginals:

\[ I(X;Y) = \sum_{x,y} p(x,y)\,\log_2 \frac{p(x,y)}{p(x)\,p(y)} \]

Everything else in this part is this expression, re-grouped. It is already Form 3 (a KL divergence). Watch it collapse into Forms 1 and 2 with nothing but log rules.

29. What has to happen first: Derivation A → Form 1 (entropies)

Ranking

30. Derivation A → Form 1 (entropies)

Worked example

\[ I(X;Y) = \sum_{x,y} p(x,y)\,\log_2 \frac{p(x,y)}{p(x)p(y)} \]

\[ = \sum_{x,y} p(x,y)\Big[\log_2 p(x,y) - \log_2 p(x)p(y)\Big] \]

\[ = \sum_{x,y} p(x,y)\log_2 p(x,y) - \sum_{x,y} p(x,y)\log_2 p(x) - \sum_{x,y} p(x,y)\log_2 p(y) \]

31. Decode the notation: Derivation A → Form 1 (entropies)

Notation

32. Plan first: Derivation A → Form 1 (finish)

Step zero

33. Derivation A → Form 1 (finish)

Worked example

\[ \sum_{x,y} p(x,y)\log_2 p(x,y) = -H(X,Y) \]

\[ \sum_{x,y} p(x,y)\log_2 p(x) = \sum_x p(x)\log_2 p(x) = -H(X) \]

\[ \sum_{x,y} p(x,y)\log_2 p(y) = -H(Y) \]

\[ \boxed{\,I(X;Y) = H(X) + H(Y) - H(X,Y)\,} \]

34. Say it in words: Derivation A → Form 1 (finish)

Translation

\( \boxed{\,I(X;Y) = H(X) + H(Y) - H(X,Y)\,} \)

35. Restore the missing line: Form 1 numerically — it's the gap we spotted

Fill the middle

36. Form 1 numerically — it's the gap we spotted

Worked example

Plug our three entropies straight into Form 1. This is the 0.2781 we flagged in Part 1, now earned:

import numpy as np

def entropy(p):
    p = np.asarray(p, dtype=float); p = p[p > 0]
    return -np.sum(p * np.log2(p))

J = np.array([[0.4, 0.1], [0.1, 0.4]])
px, py = J.sum(1), J.sum(0)
HX  = entropy(px)                     # 1.0
HY  = entropy(py)                     # 1.0
HXY = entropy(J.ravel())             # 1.7219
mi  = HX + HY - HXY
print(HX, HY, round(HXY, 4))
print("I(X;Y) =", round(mi, 4))       # 0.2781

37. Form 2: conditional entropy

Section

Part 3 of 7 — uncertainty reduced

38. Conditional entropy H(Y|X)

Concept

H(Y|X) is the average uncertainty left in Y after you already know X — the leftover surprise once the weather is revealed. It is defined by the chain rule of entropy:

\[ H(Y\mid X) = H(X,Y) - H(X) \]

Read it as: the total joint uncertainty, minus the part X already accounts for. Whatever remains is the uncertainty in Y that X couldn't explain.

39. What has to happen first: Derivation B → Form 2

Ranking

40. Derivation B → Form 2

Worked example

\[ I(X;Y) = H(X) + H(Y) - H(X,Y) \]

\[ = H(Y) + \big(H(X) - H(X,Y)\big) \]

\[ = H(Y) - \big(H(X,Y) - H(X)\big) = H(Y) - H(Y\mid X) \]

\[ \boxed{\,I(X;Y) = H(Y) - H(Y\mid X)\,} \]

41. Draw the shape of it: Derivation B → Form 2

Blank canvas

42. Form 2 numerically

Worked example

Compute H(Y|X) = H(X,Y) − H(X), then subtract from H(Y). Must land on 0.2781 again:

import numpy as np

def entropy(p):
    p = np.asarray(p, dtype=float); p = p[p > 0]
    return -np.sum(p * np.log2(p))

J = np.array([[0.4, 0.1], [0.1, 0.4]])
px, py = J.sum(1), J.sum(0)
HY   = entropy(py)                    # 1.0
HYgX = entropy(J.ravel()) - entropy(px)   # H(X,Y) - H(X) = H(Y|X)
print("H(Y)   =", round(HY, 4))       # 1.0
print("H(Y|X) =", round(HYgX, 4))     # 0.7219
print("I(X;Y) = H(Y) - H(Y|X) =", round(HY - HYgX, 4))  # 0.2781

43. Where does each piece belong: Lesson 29: Mutual Information & Contrastive…

Sorting

44. MI is symmetric

Concept

Form 1 is untouched if you swap X and Y — so I(X;Y) = I(Y;X). By the same algebra, MI also equals H(X) − H(X|Y). Weather tells you as much about umbrellas as umbrellas tell you about weather.

Verified: H(X|Y) = H(X,Y) − H(Y) = 0.7219, so H(X) − H(X|Y) = 0.2781 — identical. Information is a shared quantity, not a one-way arrow.

45. Form 3: distance from independence

Section

Part 4 of 7 — the KL view

46. KL divergence, one line

Concept

From Lesson 14, the KL divergence measures how far one distribution q is from another p, in bits:

\[ D_{KL}(p \,\|\, q) = \sum_i p_i \log_2 \frac{p_i}{q_i} \;\ge\; 0 \]

It is 0 exactly when p = q, and strictly positive otherwise. MI's definition is a KL — between the true joint p(x,y) and the pretend-independent product p(x)p(y).

47. What has to happen first: Derivation C → Form 3 (it's immediate)

Ranking

48. Derivation C → Form 3 (it's immediate)

Worked example

\[ D_{KL}\big(p(x,y)\,\|\,p(x)p(y)\big) = \sum_{x,y} p(x,y)\,\log_2 \frac{p(x,y)}{p(x)p(y)} \]

\[ \boxed{\,I(X;Y) = D_{KL}\big(p(x,y)\,\|\,p(x)p(y)\big)\,} \]

49. Decode the notation: Derivation C → Form 3 (it's immediate)

Notation

50. Guess the shape of the answer: Form 3 numerically

Estimation

51. Form 3 numerically

Worked example

Build the product p(x)p(y) with an outer product, then sum the weighted log-ratio. Third road, same 0.2781:

import numpy as np

J = np.array([[0.4, 0.1], [0.1, 0.4]])
px, py = J.sum(1), J.sum(0)
prod = np.outer(px, py)               # p(x)p(y): the independent joint
mi_kl = np.sum(J * np.log2(J / prod)) # KL(joint || product)
print("outer(px,py) =\n", prod)
print("I(X;Y) as KL =", round(mi_kl, 4))   # 0.2781

52. Which is which, by real p(x,y)

Discrimination

53. Pointwise mutual information

Concept

Inside the sum, each cell contributes p(x,y)·log₂[p(x,y)/(p(x)p(y))]. The bare log-ratio log₂[p(x,y)/(p(x)p(y))] is the pointwise MI (PMI) of one outcome pair — positive when the pair co-occurs more than chance, negative when less.

Sum the contributions: 0.27123 − 0.13219 − 0.13219 + 0.27123 = 0.2781. Individual PMIs can be negative, but their probability-weighted sum — the MI — is always ≥ 0. That guarantee is next.

54. Which is which, by PMI (bits)

Discrimination

55. Restore the missing line: All three forms agree — the master check

Fill the middle

56. All three forms agree — the master check

Worked example

Three independent derivations, one dataset, one number. This is the payoff slide — verified by execution:

import numpy as np

def entropy(p):
    p = np.asarray(p, dtype=float); p = p[p > 0]
    return -np.sum(p * np.log2(p))

J = np.array([[0.4, 0.1], [0.1, 0.4]])
px, py = J.sum(1), J.sum(0)
form1 = entropy(px) + entropy(py) - entropy(J.ravel())
form2 = entropy(py) - (entropy(J.ravel()) - entropy(px))
form3 = np.sum(J * np.log2(J / np.outer(px, py)))
print(round(form1, 6), round(form2, 6), round(form3, 6))

57. Bounds & the independence test

Section

Part 5 of 7 — I ≥ 0, I ≤ min(H), I = 0 ⟺ ⊥

58. MI is never negative

Concept

Because MI is a KL divergence and KL ≥ 0 always (Gibbs' inequality, Lesson 14), mutual information is non-negative:

\[ I(X;Y) = D_{KL}\big(p(x,y)\,\|\,p(x)p(y)\big) \;\ge\; 0 \]

You can never lose information by observing a variable — at worst it's useless (I = 0). This is why individual PMIs may go negative but the average cannot.

59. I = 0 exactly when independent

Concept

KL is 0 iff its two arguments are equal. So I(X;Y) = 0 iff p(x,y) = p(x)p(y) for every cell — which is the definition of independence:

\[ I(X;Y) = 0 \;\Longleftrightarrow\; p(x,y) = p(x)p(y)\;\;\forall x,y \;\Longleftrightarrow\; X \perp Y \]

This is the strongest independence test there is: it certifies no dependence of any kind, linear or not. Contrast that with correlation, which only sees straight lines.

60. An independent joint scores exactly 0

Worked example

Force independence by using the product p(x)p(y) as the joint. Its MI must be 0 — the KL of a distribution against itself:

import numpy as np

J = np.array([[0.4, 0.1], [0.1, 0.4]])
px, py = J.sum(1), J.sum(0)
Ind = np.outer(px, py)                # force independence
ipx, ipy = Ind.sum(1), Ind.sum(0)
mi = np.sum(Ind * np.log2(Ind / np.outer(ipx, ipy)))
print("Ind =\n", Ind)
print("I(X;Y) for the product =", round(mi, 6))   # 0.0

61. Fill in: verdict for An independent joint scores exactly 0

Comparison

62. MI is capped by each entropy

Concept

Since H(Y|X) ≥ 0, Form 2 gives I = H(Y) − H(Y|X) ≤ H(Y). By symmetry I ≤ H(X) too. So MI is squeezed between 0 and the smaller of the two entropies:

\[ 0 \;\le\; I(X;Y) \;\le\; \min\big(H(X),\,H(Y)\big) \]

For our joint: I = 0.2781 ≤ min(1.0, 1.0) = 1.0 ✓. MI hits its ceiling H(Y) only when X fully determines Y (H(Y|X) = 0) — perfect dependence.

63. Something is wrong here: zero correlation ⇒ MI = 0

Anomaly

64. Trap: zero correlation ⇒ MI = 0

Trap

65. Break it on purpose: zero correlation ⇒ MI = 0

Break the constraint

66. Restore the missing line: Y = X²: correlation ≈ 0, MI large

Fill the middle

67. Y = X²: correlation ≈ 0, MI large

Worked example

Let X be uniform on {−1, 0, 1} and Y = X². Y is a deterministic function of X, so they're maximally dependent — but the parabola is symmetric, so correlation collapses:

import numpy as np

# X uniform on {-1, 0, 1}; Y = X^2 in {0, 1}. Fully determined, yet...
Jd = np.zeros((3, 2))
Jd[0, 1] = 1/3        # X=-1 -> Y=1
Jd[1, 0] = 1/3        # X= 0 -> Y=0
Jd[2, 1] = 1/3        # X= 1 -> Y=1
px, py = Jd.sum(1), Jd.sum(0)
m = Jd > 0
mi = np.sum(Jd[m] * np.log2(Jd[m] / np.outer(px, py)[m]))

xs = np.random.default_rng(0).uniform(-1, 1, 100000)
corr = np.corrcoef(xs, xs**2)[0, 1]
print("correlation(X, X^2) =", round(corr, 4))   # ~0
print("I(X; X^2)           =", round(mi, 4), "bits")  # 0.9183

68. Application: feature selection

Section

Part 6 of 7 — rank features by MI

69. Rank features by MI with the label

Concept

A feature is useful when knowing it reduces uncertainty about the label — exactly I(feature; label). Rank features by their MI with y and keep the top ones; the rest are noise.

Unlike a correlation filter, an MI filter catches nonlinear predictors (the X²-style features a linear screen throws away). sklearn.feature_selection.mutual_info_classif estimates I for continuous features against a discrete label.

70. Predict the next row: MI finds the signal features

Pattern

71. MI finds the signal features

Worked example

Build 5 features where only columns 0 and 2 carry signal about a binary label; the other three are pure noise. Let MI expose them:

import numpy as np
from sklearn.feature_selection import mutual_info_classif

rng = np.random.default_rng(0)
n = 500
y  = rng.integers(0, 2, n)
f0 = y + 0.3*rng.normal(size=n)         # informative
f2 = 2*y - 1 + 0.3*rng.normal(size=n)   # informative
X  = np.c_[f0, rng.normal(size=n), f2, rng.normal(size=(n, 2))]
scores = mutual_info_classif(X, y, random_state=0)
print(scores.round(3))                  # [0.574 0.011 0.688 0.004 0.]

72. Watch it run: MI finds the signal features

Pattern

73. Rebuild the recipe: The mutual-information toolkit

Ranking

74. The mutual-information toolkit

Pattern

1. Definition: I(X;Y) = Σ p(x,y) log₂[p(x,y)/(p(x)p(y))] — the master expression
2. Form 1 (entropies): H(X)+H(Y)−H(X,Y) — the set-overlap
3. Form 2 (conditional): H(Y)−H(Y|X) — uncertainty reduced by seeing X
4. Form 3 (KL): D_KL(p(x,y)‖p(x)p(y)) — distance from independence
5. Bounds: 0 ≤ I ≤ min(H(X),H(Y)); I = 0 ⟺ X ⊥ Y (catches nonlinear, unlike correlation)
6. Uses: rank features by I with the label; maximize a lower bound on I via InfoNCE (CLIP)

75. Where does it stop working: The mutual-information toolkit

Edge cases

76. Check yourself — what MI = 0 means

Check

Interpret a zero. Remember MI is a KL divergence.

77. Rule out three: Check yourself — MI vs correlation

Elimination

78. Check yourself — MI vs correlation

Check

Which measure sees nonlinear structure?

79. Check yourself — the conditional form

Check

Relate MI to conditional entropy — Form 2.

80. Rule out three: Check yourself — the upper bound

Elimination

81. Check yourself — the upper bound

Check

How big can MI get? Use Form 2 and H(Y|X) ≥ 0.

82. Application: InfoNCE & CLIP

Section

Part 7 of 7 — contrastive learning

83. Learn representations by contrast

Intuition

CLIP and SimCLR don't have labels — they have pairs that belong together: an image and its caption, or two crops of the same photo. The trick is to make matched pairs' embeddings similar and mismatched pairs' embeddings different.

Doing so provably maximizes a lower bound on the mutual information between the two views. Contrastive learning is mutual-information maximization in disguise — which is why this lesson ends here.

84. The InfoNCE loss

Concept

For a true pair (x, y) among a batch of candidates {z_j} (the negatives plus the true y), InfoNCE is a softmax cross-entropy that says 'the true partner should win the similarity contest':

\[ \mathcal{L} = -\log \frac{\exp\!\big(\mathrm{sim}(x,y)/\tau\big)}{\sum_j \exp\!\big(\mathrm{sim}(x,z_j)/\tau\big)} \]

sim is cosine similarity; τ is a temperature that sharpens the softmax. Minimizing L pulls the true pair together and pushes the negatives apart — the exact cross-entropy machinery of Lesson 17, applied to representation learning.

85. What temperature τ does

Intuition

The temperature τ divides every similarity before the softmax. Small τ (like 0.1) multiplies the logits up (by 10), sharpening the softmax so it rewards the single best match harshly — the model is picky.

Large τ flattens the logits toward a uniform softmax, so the loss barely distinguishes the true partner from the negatives — the model gets lazy. CLIP tunes τ as a learnable parameter; too small overfits sharp boundaries, too large learns nothing.

In our trace τ = 0.1, so a perfect cosine of 1 becomes a logit of 10 — enormous, which is why aligned pairs drove the loss almost to zero.

86. Teach it back: What temperature τ does

Explain it

87. What has to be given first: InfoNCE traced on a tiny batch

Missing information

88. InfoNCE traced on a tiny batch

Worked example

Three image embeddings, three text embeddings, matched by index. Build the similarity matrix S = (img·txtᵀ)/τ; the correct partner for row i is column i (the diagonal):

import numpy as np

np.random.seed(0)
d = 4
img = np.random.randn(3, d)
txt = np.random.randn(3, d)
img = img / np.linalg.norm(img, axis=1, keepdims=True)   # unit vectors
txt = txt / np.linalg.norm(txt, axis=1, keepdims=True)
tau = 0.1
S = img @ txt.T / tau                                     # 3x3 logits

def softmax(r):
    e = np.exp(r - r.max()); return e / e.sum()

loss = np.mean([-np.log(softmax(S[i])[i]) for i in range(3)])
print("S =\n", S.round(2))
print("InfoNCE loss =", round(loss, 4))                  # 1.6539

89. Watch it run: InfoNCE traced on a tiny batch

Pattern

90. Guess the shape of the answer: When embeddings align, the loss collapses

Estimation

91. When embeddings align, the loss collapses

Worked example

Now make each image embedding equal its text partner (perfect alignment). The diagonal similarities dominate, softmax puts nearly all mass on the true index, and the loss drops toward zero:

import numpy as np

np.random.seed(1)
d = 4
emb = np.random.randn(3, d)
emb = emb / np.linalg.norm(emb, axis=1, keepdims=True)
tau = 0.1
S = emb @ emb.T / tau               # img and txt already agree

def softmax(r):
    e = np.exp(r - r.max()); return e / e.sum()

loss = np.mean([-np.log(softmax(S[i])[i]) for i in range(3)])
print("diagonal of S =", np.diag(S).round(1))   # [10. 10. 10.]
print("aligned loss  =", round(loss, 4))         # 0.0188

92. Work backwards from the answer: When embeddings align, the loss collapses

Reverse engineer

93. Why InfoNCE is a lower bound on MI

Concept

With a batch of N candidates, minimizing InfoNCE L maximizes a bound tying it to the mutual information between the two views x and y:

\[ I(X;Y) \;\ge\; \log_2 N - \mathcal{L} \]

Driving the loss L down pushes the lower bound on I(X;Y) up — so contrastive training literally increases the mutual information between an image and its caption. The bound tightens with bigger batches (log N), which is why CLIP trains on enormous batches of negatives.

94. By analogy: Why InfoNCE is a lower bound on MI

Analogy

95. The compression link

Concept

Information theory ties straight to compression. Shannon's source coding theorem says entropy H(X) is the minimum average bits per symbol to encode X — you cannot compress below its entropy without losing information.

Minimum description length reframes learning as compression: the best model encodes the data in the fewest bits. Lower entropy and higher MI mean more exploitable structure — a model that finds high-MI features has literally found something compressible.

96. Your turn: compute MI

Section

Project — MI from scratch

97. Project: mutual information from a joint

Concept

Build MI from a joint distribution by two of the three formulas, confirm they match, then confirm an independent joint gives exactly 0. You've derived every piece — now assemble it yourself.

Build rules: type every line yourself, mask out zero probabilities in the entropy (log 0 is undefined), and use log2 so the answer is in bits.

98. Break it if you can: Project: mutual information from a joint

Counterexample

99. Milestone 1 — marginals & entropies

Worked example

Your turn: get the marginals and their entropies from the joint. Predict H(X) for a 50/50 marginal before you print.

Hint: px = J.sum(1), py = J.sum(0); entropy(p) = −Σ p log₂ p over p > 0.

import numpy as np

def entropy(p):
    p = np.asarray(p, dtype=float); p = p[p > 0]
    return -np.sum(p * np.log2(p))

J = np.array([[0.4, 0.1], [0.1, 0.4]])
px, py = J.sum(1), J.sum(0)
print(px, py)                                  # [0.5 0.5] [0.5 0.5]
print(round(entropy(px), 4), round(entropy(py), 4))   # 1.0 1.0

100. Milestone 2 — MI two ways

Worked example

Your turn: compute MI by the entropy definition (Form 1) and by the KL form (Form 3). Predict: do they match?

Hint: Form 1 is entropy(px)+entropy(py)−entropy(J.ravel()); Form 3 is np.sum(J*np.log2(J/np.outer(px,py))).

import numpy as np

def entropy(p):
    p = np.asarray(p, dtype=float); p = p[p > 0]
    return -np.sum(p * np.log2(p))

J = np.array([[0.4, 0.1], [0.1, 0.4]])
px, py = J.sum(1), J.sum(0)
mi_def = entropy(px) + entropy(py) - entropy(J.ravel())
mi_kl  = np.sum(J * np.log2(J / np.outer(px, py)))
print(round(mi_def, 4), round(mi_kl, 4))       # 0.2781 0.2781

101. Milestone 3 — independence gives 0

Worked example

Your turn: build an independent joint with np.outer(px, py) and confirm its MI is 0 — the proof of the I = 0 ⟺ independent test.

Hint: for Ind = np.outer(px, py), the joint already equals the product, so every log-ratio is log(1) = 0.

import numpy as np

J = np.array([[0.4, 0.1], [0.1, 0.4]])
px, py = J.sum(1), J.sum(0)
Ind = np.outer(px, py)                         # independent joint
ipx, ipy = Ind.sum(1), Ind.sum(0)
mi = np.sum(Ind * np.log2(Ind / np.outer(ipx, ipy)))
print(round(mi, 6))                            # 0.0

102. What each one costs: Milestone 3 — independence gives 0

Trade off

103. The full program

Concept

import numpy as np

def entropy(p):
    p = np.asarray(p, dtype=float); p = p[p > 0]
    return -np.sum(p * np.log2(p))

def mutual_information(J):
    px, py = J.sum(1), J.sum(0)
    return np.sum(J * np.log2(J / np.outer(px, py)))   # = H(X)+H(Y)-H(X,Y)

J = np.array([[0.4, 0.1], [0.1, 0.4]])
px, py = J.sum(1), J.sum(0)
print('MI(dependent)   =', round(mutual_information(J), 4))            # 0.2781
print('MI(independent) =', round(mutual_information(np.outer(px, py)), 6))  # 0.0

If MI agrees across the formulas and zeroes out for an independent joint — you can measure dependence the way representation learning does.

104. Fill in: value for The full program

Comparison

105. Show it off

Concept

Out loud, slides closed: explain (1) the three roads from the definition to Forms 1, 2, and 3, (2) why zero MI is strictly stronger than zero correlation, and (3) what InfoNCE pulls together and pushes apart.

Stretch (homework): prove I(X;Y) ≥ 0 from D_KL ≥ 0, and use mutual_info_classif to rank the features of a 100-feature dataset. Mutual information returns in CLIP and contrastive pre-training (Week 47).

106. Connect it up: Lesson 29: Mutual Information & Contrastive Learning

Connect it up

107. What you can do now

Recap

• Read entropy H(X) = −Σ p log₂ p as bits of uncertainty, masking zeros
• Derive MI three equivalent ways — H(X)+H(Y)−H(X,Y), H(Y)−H(Y|X), D_KL(p(x,y)‖p(x)p(y)) — and get 0.2781 bits three times on one joint
• Prove I ≥ 0, I ≤ min(H(X),H(Y)), and I = 0 ⟺ independence, then show Y = X² has correlation ≈ 0 but MI 0.9183
• Rank features by mutual information with the label using mutual_info_classif
• Trace the InfoNCE loss (random 1.6539 → aligned 0.0188) and connect entropy to compression (MDL)