Lesson 73: Metric Learning, Siamese Networks & Triplet Loss

1. Metric Learning & Siamese Networks

Title

USAAIO · Lesson 73

Learn an embedding space where distance = semantic distance: triplet loss, contrastive loss, hard negative mining, and why it connects to the attention you built in the transformer.

2. By the end of this lesson you can

Objectives

1. Explain metric learning: optimize an embedding so distance reflects similarity
2. Build a Siamese network that maps two inputs through shared weights and compares embeddings
3. Implement triplet loss: pull anchor-positive together, push anchor-negative apart with margin m
4. Implement contrastive loss and identify when each loss is appropriate
5. Apply online hard negative mining and explain why it accelerates convergence

3. What survived from Phase 2 Algorithm Review?

Warm-up

4. What is metric learning?

Section

Part 1 of 3

5. Learned distance, not raw distance

Concept

Raw Euclidean distance in pixel space is meaningless: two photos of the same person under different lighting are far apart; two different people in matching poses may be close.

Metric learning trains a function f(x) such that ‖f(x_a) − f(x_p)‖ < ‖f(x_a) − f(x_n)‖ for same-class pairs (a, p) and different-class pairs (a, n).

6. Fill in: same-person distance for Learned distance, not raw distance

Comparison

7. Siamese network architecture

Concept

A Siamese network is two copies of the same network (shared weights) applied to two inputs. The outputs are compared with a distance function — there is no softmax over fixed classes.

• Both branches = EmbeddingNet(x) with tied parameters
• Output: embedding vectors (e.g. 4-d or 128-d), L2-normalized
• Loss computed on pairs or triplets, not individual labels

Shared weights are the key: each sample routes through identical f, so similarity is measured in a consistent space.

8. Break it if you can: Siamese network architecture

Counterexample

9. Why L2-normalize the output?

Intuition

Project every embedding onto the unit hypersphere. Then ‖ea − eb‖² = 2 − 2·cos(θ) — Euclidean distance is a monotone function of angle. The network learns to spread identities across the sphere.

Without normalization the network can cheat by growing embedding magnitudes without moving the angular directions — you'd be measuring the scale of activations, not semantic similarity.

10. By analogy: Why L2-normalize the output?

Analogy

11. Triplet loss & contrastive loss

Section

Part 2 of 3

12. Triplet loss: anchor, positive, negative

Concept

A triplet (a, p, n): anchor and positive are the same class; anchor and negative are different classes. The loss pushes d_ap below d_an by at least margin m.

\[ \mathcal{L}_{\text{trip}} = \max\!\bigl(0,\; d(a,p) - d(a,n) + m\bigr) \]

When d_ap < d_an − m the triplet is satisfied and contributes zero loss. Only violated triplets (where the negative is too close) drive gradients.

13. Teach it back: Triplet loss: anchor, positive, negative

Explain it

14. Guess the shape of the answer: Triplet loss — step by step

Estimation

15. Triplet loss — step by step

Worked example

Use the trained EmbeddingNet (4-d, L2-normalized). Compute loss for 3 sampled triplets with margin m=0.5.

import torch, torch.nn as nn, torch.nn.functional as F
import numpy as np
from sklearn.datasets import make_blobs

np.random.seed(42); torch.manual_seed(0)
X, y = make_blobs(150, 8, centers=3, cluster_std=4.0, random_state=42)
X = ((X - X.mean(0)) / (X.std(0)+1e-6)).astype(np.float32)
Xt = torch.tensor(X)

class EmbedNet(nn.Module):
    def __init__(self): super().__init__(); self.net = nn.Sequential(nn.Linear(8,16), nn.ReLU(), nn.Linear(16,4))
    def forward(self, x): return F.normalize(self.net(x), dim=-1)

model = EmbedNet()
# triplets (a, p, n) indices sampled with rng seed 10
triplets = [(118, 106, 63), (63, 141, 8), (67, 103, 15)]
print(f"{'triplet':>8}  {'d_ap':>8}  {'d_an':>8}  {'loss':>8}")
for ai, pi, ni in triplets:
    ea, ep, en = model(Xt[ai:ai+1]), model(Xt[pi:pi+1]), model(Xt[ni:ni+1])
    dap = torch.norm(ea-ep).item(); dan = torch.norm(ea-en).item()
    loss = max(0.0, dap - dan + 0.5)
    print(f"{str((ai,pi,ni)):>8}  {dap:.4f}    {dan:.4f}    {loss:.4f}")

16. What each one costs: Triplet loss — step by step

Trade off

17. Contrastive loss: pair-based alternative

Concept

Contrastive loss operates on pairs (x_i, x_j) with a binary label y=1 (same class) or y=0 (different). Minimize distance for positives; push negatives past margin m.

\[ \mathcal{L}_{\text{contra}} = y\,d^2 + (1-y)\,\max(0,\, m - d)^2 \]

18. Guess the shape of the answer: Contrastive loss — two pairs

Estimation

19. Contrastive loss — two pairs

Worked example

Compute contrastive loss for a same-class pair and a different-class pair after 300 epochs of training (margin m=1.0).

# After 300-epoch training, embeddings from model_rand
# pair (0,1): same class (label=1)
# pair (0,2): different class (label=0)
# model_rand embeddings (L2-normalized 4-d)
with torch.no_grad():
    embs = model(Xt[:3])         # shape (3, 4)
    d01 = torch.norm(embs[0]-embs[1]).item()
    d02 = torch.norm(embs[0]-embs[2]).item()

margin = 1.0
loss_same = d01**2               # y=1
loss_diff = max(0, margin-d02)**2  # y=0
print(f'd(0,1)={d01:.4f}  L_same={loss_same:.4f}')
print(f'd(0,2)={d02:.4f}  L_diff={loss_diff:.4f}')

20. Work backwards from the answer: Contrastive loss — two pairs

Reverse engineer

21. Something is wrong here: confusing triplet and contrastive loss inputs

Anomaly

22. Trap: confusing triplet and contrastive loss inputs

Trap

23. Break it on purpose: confusing triplet and contrastive loss inputs

Break the constraint

24. Hard negative mining

Section

Part 3 of 3

25. Why most triplets are useless

Concept

After a few epochs, the vast majority of randomly sampled triplets are already satisfied (d_ap < d_an − m). They contribute zero gradient and zero learning — you're burning compute on easy examples.

Hard negative mining selects the most informative negatives: the negative closest to the anchor in the current embedding space. These are the ones the model is most confused by.

26. Hard mining strategy: online vs offline

Concept

Offline mining: run inference on all data before each epoch, compute all pairwise distances, pick the hardest triplets globally. Expensive but thorough.

Online mining: within each mini-batch, find the hardest positive (farthest same-class) and hardest negative (closest diff-class) on the fly. Standard in modern face recognition (FaceNet).

• Online hard positive: argmax_{p: y_p=y_a} ‖f(a)−f(p)‖ within batch
• Online hard negative: argmin_{n: y_n≠y_a} ‖f(a)−f(n)‖ within batch
• Semi-hard negative: negative farther than positive but within margin (avoids collapse)

27. Guess the shape of the answer: Random vs hard negatives — training…

Estimation

28. Random vs hard negatives — training comparison

Worked example

Train two identical Siamese nets (8-d → 16 → 4-d, L2-norm) for 300 epochs on a 3-identity benchmark. Only the negative-sampling strategy differs.

import torch, torch.nn as nn, torch.nn.functional as F, numpy as np
from sklearn.datasets import make_blobs
np.random.seed(42); torch.manual_seed(0)
X, y = make_blobs(150,8,centers=3,cluster_std=4.0,random_state=42)
X = ((X-X.mean(0))/(X.std(0)+1e-6)).astype(np.float32); Xt=torch.tensor(X)
class EmbedNet(nn.Module):
    def __init__(self): super().__init__(); self.net=nn.Sequential(nn.Linear(8,16),nn.ReLU(),nn.Linear(16,4))
    def forward(self,x): return F.normalize(self.net(x),dim=-1)
def samp(rng,hard=False,embs=None):
    ai=int(rng.integers(150)); lbl=y[ai]
    pp=np.where(y==lbl)[0]; pp=pp[pp!=ai]; np2=np.where(y!=lbl)[0]
    if hard and embs is not None:
        pi=int(pp[np.argmax(np.linalg.norm(embs[pp]-embs[ai],axis=1))])
        ni=int(np2[np.argmin(np.linalg.norm(embs[np2]-embs[ai],axis=1))])
    else: pi=int(rng.choice(pp)); ni=int(rng.choice(np2))
    return ai,pi,ni
def train(hard,seed=0):
    torch.manual_seed(seed); rng=np.random.default_rng(seed)
    m=EmbedNet(); opt=torch.optim.Adam(m.parameters(),lr=5e-3); losses=[]
    for _ in range(300):
        embs=m(Xt).detach().numpy() if hard else None
        opt.zero_grad(); total=torch.tensor(0.0)
        for __ in range(16):
            ai,pi,ni=samp(rng,hard=hard,embs=embs)
            ea,ep,en=m(Xt[ai:ai+1]),m(Xt[pi:pi+1]),m(Xt[ni:ni+1])
            total+=torch.clamp(torch.norm(ea-ep)-torch.norm(ea-en)+0.5,min=0)
        (total/16).backward(); opt.step(); losses.append((total/16).item())
    return m,losses
model_rand,lr=train(False); model_hard,lh=train(True)
print('rand ep1:',round(lr[0],4),'ep50:',round(lr[49],4),'ep150:',round(lr[149],4))
print('hard ep1:',round(lh[0],4),'ep50:',round(lh[49],4),'ep150:',round(lh[149],4))

29. Fill in: epoch 10 for Random vs hard negatives — training…

Comparison

30. Embedding quality: inter/intra distance ratio

Concept

After 300 epochs, measure the mean intra-class distance (same identity) and mean inter-class distance (different identity) across all pairs.

Hard mining reduces intra-class spread more aggressively (0.30 → 0.19) while maintaining inter-class separation, improving the ratio by 1.5×.

31. By analogy: Embedding quality: inter/intra distance ratio

Analogy

32. Something is wrong here: always using the absolute hardest negative

Anomaly

33. Trap: always using the absolute hardest negative

Trap

34. Which of these survive contact with Lesson 73: Metric Learning, Siamese Networks…?

Two truths and a lie

35. Connection to attention (Lesson 65 callback)

Concept

Both metric learning and attention compute similarity scores between vectors. In dot-product attention (Lesson 65): scores = QK^T / √d — the raw score is a cosine similarity between L2-normalized query and key.

\[ \text{Attention}(Q,K,V) = \text{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V \]

If Q and K are L2-normalized embeddings, Q·K^T is exactly the cosine similarity matrix — attention weights are metric-learning similarity scores routed through a softmax.

36. Break it if you can: Connection to attention (Lesson 65 callback)

Counterexample

37. Rebuild the recipe: The metric learning recipe

Ranking

38. The metric learning recipe

Pattern

1. Architecture: Siamese EmbeddingNet with shared weights; L2-normalize output
2. Triplets: sample (anchor a, positive p, negative n); loss = max(0, d_ap − d_an + m)
3. Contrastive: pairs with label y; loss = y·d² + (1−y)·max(0, m−d)²
4. Hard mining: closest same-class positive + closest diff-class negative per batch
5. Evaluate: inter/intra distance ratio (higher = better separation)
6. Connection: L2-normalized embedding dot product = cosine sim = attention score

39. Where does it stop working: The metric learning recipe

Edge cases

40. Rule out three: Check yourself — triplet loss

Elimination

41. Check yourself — triplet loss

Check

Work it out before clicking.

42. Answer it before you see the options: Check yourself — hard negative mining

Prediction

43. Check yourself — hard negative mining

Check

Think about what 'hard' means.

44. Rule out three: Check yourself — metric learning & attention

Elimination

45. Check yourself — metric learning & attention

Check

Connect the two ideas.

46. Your turn: Siamese face verifier

Section

Project

47. Project: Siamese EmbeddingNet with triplet loss

Concept

Build a Siamese network that learns a 4-d embedding space for a 3-identity dataset. Train it twice — once with random negatives and once with hard negatives — and compare the inter/intra distance ratio.

Build rules: L2-normalize outputs with F.normalize(out, dim=-1). Never pass a satisfied triplet's gradient through — use torch.clamp(..., min=0). Verify inter/intra ratio improves with hard mining.

48. Milestone 1 — EmbeddingNet + triplet forward pass

Worked example

Your turn: define EmbeddingNet (8→16→4, ReLU, L2-norm) and compute the triplet loss for one (a, p, n) tuple by hand.

Hint: F.normalize(self.net(x), dim=-1) returns unit-norm vectors. torch.norm(ea-ep) gives L2 distance. Loss = torch.clamp(dap - dan + margin, min=0).

import torch, torch.nn as nn, torch.nn.functional as F
import numpy as np
from sklearn.datasets import make_blobs

np.random.seed(42); torch.manual_seed(0)
X, y = make_blobs(150, 8, centers=3, cluster_std=4.0, random_state=42)
X = ((X - X.mean(0))/(X.std(0)+1e-6)).astype(np.float32)
Xt = torch.tensor(X)

class EmbedNet(nn.Module):
    def __init__(self): super().__init__(); self.net = nn.Sequential(nn.Linear(8,16), nn.ReLU(), nn.Linear(16,4))
    def forward(self, x): return F.normalize(self.net(x), dim=-1)

model = EmbedNet()
ai, pi, ni = 118, 106, 63   # anchor, pos, neg
ea, ep, en = model(Xt[ai:ai+1]), model(Xt[pi:pi+1]), model(Xt[ni:ni+1])
dap = torch.norm(ea-ep).item(); dan = torch.norm(ea-en).item()
loss = torch.clamp(torch.tensor(dap - dan + 0.5), min=0).item()
print(f'dap={dap:.4f}  dan={dan:.4f}  loss={loss:.4f}')
print('L2 norm ea:', round(torch.norm(ea).item(), 4))

49. Milestone 2 — training loop, random negatives

Worked example

Your turn: write the 300-epoch loop with 16 random triplets per step. Predict where the loss plateaus.

Hint: opt.zero_grad() once per epoch (outside the inner loop); accumulate losses across the 16 triplets, then divide and .backward(). Use Adam(lr=5e-3).

opt = torch.optim.Adam(model.parameters(), lr=5e-3)
rng = np.random.default_rng(0)

def sample_triplet(X, y, rng):
    ai = int(rng.integers(len(X))); lbl = y[ai]
    pos = np.where(y==lbl)[0]; pos = pos[pos!=ai]
    neg = np.where(y!=lbl)[0]
    return ai, int(rng.choice(pos)), int(rng.choice(neg))

for epoch in range(300):
    opt.zero_grad(); total = torch.tensor(0.0)
    for _ in range(16):
        ai,pi,ni = sample_triplet(X, y, rng)
        ea,ep,en = model(Xt[ai:ai+1]),model(Xt[pi:pi+1]),model(Xt[ni:ni+1])
        total += torch.clamp(torch.norm(ea-ep)-torch.norm(ea-en)+0.5, min=0)
    (total/16).backward(); opt.step()
print('final loss:', round((total/16).item(), 4))

50. What each one costs: Milestone 2 — training loop, random negatives

Trade off

51. Milestone 3 — hard negative mining + ratio comparison

Worked example

Your turn: re-train with hard negatives (snapshot embeddings each epoch, pick argmin diff-class distance). Predict whether hard mining reaches a higher or lower inter/intra ratio.

Hint: embs_np = model(Xt).detach().numpy() at the top of each epoch. For hard neg: neg_pool = where(y!=y_a), neg_idx = neg_pool[argmin(‖embs_np[neg_pool] − embs_np[a_i]‖)].

torch.manual_seed(0); model_h = EmbedNet()
opt_h = torch.optim.Adam(model_h.parameters(), lr=5e-3)
rng2 = np.random.default_rng(0)

for epoch in range(300):
    with torch.no_grad(): embs_np = model_h(Xt).numpy()
    opt_h.zero_grad(); total = torch.tensor(0.0)
    for _ in range(16):
        ai = int(rng2.integers(len(X))); lbl = y[ai]
        pos_pool = np.where(y==lbl)[0]; pos_pool = pos_pool[pos_pool!=ai]
        neg_pool = np.where(y!=lbl)[0]
        pi = int(pos_pool[np.argmax(np.linalg.norm(embs_np[pos_pool]-embs_np[ai], axis=1))])
        ni = int(neg_pool[np.argmin(np.linalg.norm(embs_np[neg_pool]-embs_np[ai], axis=1))])
        ea,ep,en = model_h(Xt[ai:ai+1]),model_h(Xt[pi:pi+1]),model_h(Xt[ni:ni+1])
        total += torch.clamp(torch.norm(ea-ep)-torch.norm(ea-en)+0.5, min=0)
    (total/16).backward(); opt_h.step()
print('hard final loss:', round((total/16).item(), 4))

52. Fill in: inter d for Milestone 3 — hard negative mining + ratio…

Comparison

53. Show it off

Concept

Out loud, slides closed: (1) explain the triplet loss formula and why satisfied triplets contribute zero gradient, (2) describe what hard negative mining selects and why it can cause collapse without a semi-hard guard, (3) state the connection between L2-normalized embeddings and attention scores.

Stretch (homework): implement face verification with a threshold on d(f(x1), f(x2)) and plot ROC at various thresholds; compare contrastive vs triplet loss on the same 3-identity data; analyze how metric learning relates to attention using the similarity matrix Q·K^T from Lesson 65.

54. Connect it up: Lesson 73: Metric Learning, Siamese Networks & Triplet Loss

Connect it up

55. What you can do now

Recap

• Metric learning: train f(x) so distance = semantic similarity
• Siamese network: shared-weight branches; L2-normalize output onto unit sphere
• Triplet loss: max(0, d_ap − d_an + m); zero for satisfied triplets
• Contrastive loss: y·d² + (1−y)·max(0,m−d)²; pair-based alternative
• Hard mining: argmin diff-class distance; improves inter/intra ratio; semi-hard avoids collapse
• Attention link: L2-normalized Q·K^T = cosine similarity = metric learning score