Robust SIFT + RANSAC Landmark Matching (Before/After Registration)

Discussion prompt

We want a stable correspondence set between the same skin region photographed weeks apart, then a geometric model that maps Visit 1 to Visit 2 so lesions can be measured.

That is stated as though it always holds. Do one of two things: produce a case where it fails, or say precisely what rules such a case out. "It just does" is not on the menu.

Hint: Hunt at the extremes first — zero, one, negative, empty, equal. If every extreme survives, the reason they survive is the proof.

Match the pairs

From Today's four levers — match each one to what it actually does. The descriptions have been shuffled.

• c1. Diagnose
• c2. RootSIFT
• c3. Cross-check
• c4. MAGSAC++

• b1. Localize the failing stage before touching a knob.
• b2. Right metric for gradient histograms.
• b3. Mutual NN purifies the match set.
• b4. σ-consensus instead of one hard threshold.

Why: Diagnose, RootSIFT, Cross-check, MAGSAC++ are easy to tell apart while they are sitting next to their descriptions and much harder afterwards, which is what this checks.

Discussion prompt

Explain The pipeline is four stages by analogy to something with no Computer Vision in it at all — a queue, a recipe, a map, a bank balance, whatever fits. Then say where your analogy breaks.

Hint: An analogy that never breaks is not an analogy, it is the same idea wearing a hat. Find the seam — that is the part that is actually new.

Discussion prompt

Explain Four distinct failure modes to a student a year behind you. No notation, no jargon they have not met — and it still has to be true.

Hint: If your explanation needs a symbol they have never seen, you are describing the notation rather than the idea.

Comparison matrix

From Instrument every stage: refill the Stage column from what you know. The rest of the table is as it appeared.

Comparison matrix

From Reading a real diagnostic run: every row here is a choice with a cost. Fill the Verdict column, then say which row you would actually pick and what you give up for it.

Predict first

A student writes this, and it looks reasonable:

Matches look bad, so loosen everything at once: ratio → 0.9, RANSAC threshold → 10.

It is wrong. Say what breaks — and say it before you turn the page.

Correct: You get more 'matches' and more 'inliers', so it feels better — but you have masked the failing stage and inflated the outlier rate fed to the estimator.

Print the per-stage funnel first: keypoints, raw, ratio survivors, inlier ratio, reprojection error.

Why: You get more 'matches' and more 'inliers', so it feels better — but you have masked the failing stage and inflated the outlier rate fed to the estimator.

Put in order

These are the steps of Recipe: diagnosis-first, scrambled. Put them back in order before the next slide shows you.

1. Print the funnel — the five stage metrics, every run.
2. Find the first wrong number — the earliest stage that collapses is the cause.
3. Change one thing that targets that stage.
4. Re-read the funnel — confirm that number moved and the rest held.
5. Keep an ablation log — config → correct-match rate, so claims are evidence, not vibes.

Why: This is the order the recipe itself gives. Recalling the sequence without the slide in front of you is the difference between recognising the method and being able to run it — most of what goes wrong in practice is a step done out of turn.

Predict first

A student writes this, and it looks reasonable:

RootSIFT the main descriptors, but leave the augmented stable-spot descriptors as raw SIFT, then vstack and match in one FLANN index.

It is wrong. Say what breaks — and say it before you turn the page.

Correct: Their scales differ by orders of magnitude (unit-norm vs raw counts).

Apply to_rootsift to the spot descriptors before the vstack.

Why: Their scales differ by orders of magnitude (unit-norm vs raw counts). The L2 distance is meaningless, so the nearest neighbor is essentially random.

Eliminate the wrong options

You apply RootSIFT to the main detector but forget it inside augment_with_stable_spot_keypoints, so the extra mole keypoints keep raw SIFT descriptors. The two sets are vstacked and matched with one FLANN index. What happens?

3 of these 4 are wrong. Strike them one at a time, and say what rules each one out before you strike the next. The survivor is the answer.

• A. The mole keypoints rarely match correctly — Euclidean distances between RootSIFT and raw-SIFT vectors aren't comparable, silently losing the most stable landmarks.
• B. Nothing — FLANN L2-normalizes descriptors internally, so the two spaces are reconciled before search.
• C. The program raises a dimension-mismatch error because RootSIFT changes the descriptor length.
• D. Matching improves, because raw SIFT preserves more magnitude information than RootSIFT.

Survives elimination: A

Why: RootSIFT is an element-wise change of coordinates (L1-normalize then sqrt), leaving the descriptor 128-D but unit-L2-norm. Raw SIFT rows have arbitrary, much larger magnitudes. FLANN's kd-tree searches the raw stacked vectors with no normalization, so a unit-norm query and a large-magnitude train vector produce a meaningless distance — the mole keypoints match essentially at random and drop out as outliers, with no exception raised.

Check your understanding

You apply RootSIFT to the main detector but forget it inside augment_with_stable_spot_keypoints, so the extra mole keypoints keep raw SIFT descriptors. The two sets are vstacked and matched with one FLANN index. What happens?

• A. The mole keypoints rarely match correctly — Euclidean distances between RootSIFT and raw-SIFT vectors aren't comparable, silently losing the most stable landmarks. (correct)
• B. Nothing — FLANN L2-normalizes descriptors internally, so the two spaces are reconciled before search.
• C. The program raises a dimension-mismatch error because RootSIFT changes the descriptor length.
• D. Matching improves, because raw SIFT preserves more magnitude information than RootSIFT.

Answer: A

Why: RootSIFT is an element-wise change of coordinates (L1-normalize then sqrt), leaving the descriptor 128-D but unit-L2-norm. Raw SIFT rows have arbitrary, much larger magnitudes. FLANN's kd-tree searches the raw stacked vectors with no normalization, so a unit-norm query and a large-magnitude train vector produce a meaningless distance — the mole keypoints match essentially at random and drop out as outliers, with no exception raised.

Why B tempts people

FLANN's kd-tree performs no normalization; it indexes and searches the descriptor rows exactly as given.

Why C tempts people

RootSIFT preserves dimensionality (128-D in, 128-D out); the vstack succeeds, so there is no shape error — which is what makes the failure silent.

Why D tempts people

Those magnitudes are precisely what lighting corrupts; RootSIFT discards large-bin dominance on purpose, and mixing spaces destroys comparability rather than preserving useful information.

Discussion prompt

For each query descriptor we take its two nearest neighbors and keep the match only if the best is decisively closer than the runner-up — Lowe's ratio test.

Suppose that were not true. What is the first thing in Robust SIFT + RANSAC Landmark Matching (Before/After Registration) that would stop working?

Hint: Follow it one step downstream. The answer is whatever was quietly relying on it.

Sort into buckets

Every line below is part of the definition of Localization principle or of Mutual (symmetric) NN — one or the other, never both. Put each where it belongs.

Localization principle

Each stage emits a count you can read.; Find the first stage whose number is wrong; Everything downstream is a symptom, not the cause.

Mutual (symmetric) NN

Keep a match a→b only if, matching in the reverse direction, b's nearest neighbor is a.; The agreement must be two-sided.

b1

Each stage emits a count you can read. Find the first stage whose number is wrong; tune only that stage. Everything downstream is a symptom, not the cause.

b2

Keep a match a→b only if, matching in the reverse direction, b's nearest neighbor is a. The agreement must be two-sided.

Comparison matrix

From The full ablation: refill the correct % column from what you know. The rest of the table is as it appeared.

Step through it

Step through Why precision beats recall for RANSAC one row at a time. What is driving the change, and what would the row after the last one be?

1. Step 1: inlier ratio w is 0.14
2. Step 2: inlier ratio w is 0.59
3. Step 3: inlier ratio w is 0.70

Predict first

A student writes this, and it looks reasonable:

Build the reverse map, then keep m where best_back[m.queryIdx] == m.trainIdx.

It is wrong. Say what breaks — and say it before you turn the page.

Correct: queryIdx indexes des1, but the reverse search keys are des2 indices.

Keep m where best_back[m.trainIdx] == m.queryIdx.

Why: queryIdx indexes des1, but the reverse search keys are des2 indices. You look up the wrong table entry for almost every match.

Discussion prompt

The rule this trap just fixed:

The reverse match's queryIdx is the des2 point; key the map by m.trainIdx and require it points back to m.queryIdx. Symmetry is enforced on the correct indices.

Now break it on purpose. Build a case that violates it and follow the consequences until something visibly fails. Where does the failure first show up — and would you have noticed it if you had not been looking?

Hint: The dangerous rules are the ones whose violation still produces an answer. If yours fails loudly, try to find one that fails quietly.

Predict first

A student writes this, and it looks reasonable:

Too few matches survive, so push lowe_ratio to 0.9.

It is wrong. Say what breaks — and say it before you turn the page.

Correct: On repetitive skin the 1st and 2nd neighbors are near-duplicates; 0.9 admits exactly the ambiguous matches that are most often geometrically wrong.

Keep the ratio tight; recover recall with cross-check, more keypoints, and the spatial prior.

Why: On repetitive skin the 1st and 2nd neighbors are near-duplicates; 0.9 admits exactly the ambiguous matches that are most often geometrically wrong.

Predict first

On self-similar skin the ratio test alone returns 86 'good' matches but only ~12% are geometrically correct. Adding a mutual nearest-neighbor cross-check drops it to 10 matches at ~90% correct. Why is this usually the right trade for a RANSAC-based estimator?

Answer it in your own words, now, with nothing to choose from. The options are on the next slide — and picking the right one off a list is an easier skill than producing it.

Correct: RANSAC's required iterations grow steeply as the inlier ratio falls, so trading recall for a far higher inlier ratio makes robust estimation faster and more reliable.

Why: The sample count k = log(1−p)/log(1−w^s) blows up as the inlier ratio w drops; for a homography s=4, so w from 0.12 to 0.90 changes w⁴ by ~600×. A smaller but far purer set makes a clean minimal sample likely within a handful of iterations and yields a more accurate fit. Trading recall for precision is therefore exactly what a robust estimator wants.

Check your understanding

On self-similar skin the ratio test alone returns 86 'good' matches but only ~12% are geometrically correct. Adding a mutual nearest-neighbor cross-check drops it to 10 matches at ~90% correct. Why is this usually the right trade for a RANSAC-based estimator?

• A. RANSAC's required iterations grow steeply as the inlier ratio falls, so trading recall for a far higher inlier ratio makes robust estimation faster and more reliable. (correct)
• B. Cross-check increases the absolute number of true correspondences, which always improves the homography.
• C. Fewer matches reduce memory use, which is the dominant cost of findHomography.
• D. The ratio test and cross-check are redundant, so removing 88% of matches has no real effect on the result.

Answer: A

Why: The sample count k = log(1−p)/log(1−w^s) blows up as the inlier ratio w drops; for a homography s=4, so w from 0.12 to 0.90 changes w⁴ by ~600×. A smaller but far purer set makes a clean minimal sample likely within a handful of iterations and yields a more accurate fit. Trading recall for precision is therefore exactly what a robust estimator wants.

Why B tempts people

Cross-check can only remove matches; recall drops from 86 to 10. It cannot create true correspondences — the benefit is the higher inlier ratio, not a larger count.

Why C tempts people

findHomography's cost is dominated by sampling iterations, not by storing a few hundred 2-D points; memory is irrelevant here.

Why D tempts people

They filter different failure modes — ambiguity (ratio) vs asymmetry (mutual NN). The 12% → 90% precision jump shows the removed 88% were mostly wrong, so the effect is large, not negligible.

Predict first

Keypoint counts are healthy on both images (thousands each) and raw match count is high, but RANSAC returns an inlier ratio of 0.03 and findHomography's average reprojection error is large. Which stage is the prime suspect?

Commit to an answer, then rate it — certain, fairly sure, or guessing — and write the rating down before you turn the page.

Correct: Matching: many descriptor matches form but few are mutually consistent, so the correspondence set is contaminated before geometry.

Why: A 3% inlier ratio with large reprojection error, despite healthy keypoint and raw-match counts, is the signature of a contaminated match set: descriptors are being paired but the pairs are mostly geometrically inconsistent. The fix is upstream of the estimator — descriptor quality (RootSIFT) and match purity (cross-check) — not a different RANSAC threshold.

The rating matters as much as the answer: confident-and-wrong is the combination that survives revision, because nothing about it feels like it needs revisiting.

Check your understanding

Keypoint counts are healthy on both images (thousands each) and raw match count is high, but RANSAC returns an inlier ratio of 0.03 and findHomography's average reprojection error is large. Which stage is the prime suspect?

• A. Matching: many descriptor matches form but few are mutually consistent, so the correspondence set is contaminated before geometry. (correct)
• B. Detection: too few keypoints are being found on smooth skin.
• C. Grayscale/CLAHE preprocessing is producing empty images.
• D. The estimator: MAGSAC is mathematically incapable of fitting a homography to skin.

Answer: A

Why: A 3% inlier ratio with large reprojection error, despite healthy keypoint and raw-match counts, is the signature of a contaminated match set: descriptors are being paired but the pairs are mostly geometrically inconsistent. The fix is upstream of the estimator — descriptor quality (RootSIFT) and match purity (cross-check) — not a different RANSAC threshold.

Why B tempts people

The premise states thousands of keypoints on both images, so detection is not starved — that hypothesis is already falsified by the funnel.

Why C tempts people

Failed preprocessing would collapse keypoint and raw-match counts to near zero; both are stated to be high, so the images are not empty.

Why D tempts people

A 3% inlier ratio means the inputs are mostly wrong; no estimator can recover structure that isn't in the correspondences. The defect is in the set fed to it, not the estimator.

Sort into buckets

Sort these by MAGSAC err (px), from memory, without looking back at Threshold-insensitivity, measured. Telling them apart on the spot is the skill; the table is only where the answer happens to be written down.

0.178

1.0; 2.0

0.181

3.0; 5.0; 8.0

g1

MAGSAC err (px) is "0.178" for 1.0, 2.0 — that is what the table on "Threshold-insensitivity, measured" records, and it is the single property separating this group from the rest.

g2

MAGSAC err (px) is "0.181" for 3.0, 5.0, 8.0 — that is what the table on "Threshold-insensitivity, measured" records, and it is the single property separating this group from the rest.

Discussion prompt

Explain Skin is not a rigid plane to a student a year behind you. No notation, no jargon they have not met — and it still has to be true.

Hint: If your explanation needs a symbol they have never seen, you are describing the notation rather than the idea.

Predict first

A student writes this, and it looks reasonable:

The homography looks loose, so drop --ransac-threshold to 1.0 to 'keep only tight inliers'.

It is wrong. Say what breaks — and say it before you turn the page.

Correct: A threshold at the noise scale rejects genuine inliers that happen to sit just outside: 77 of 200 kept, corner error ~1.0 px — looser, not stricter.

Let MAGSAC marginalize over σ; pass the threshold as a loose upper bound.

Why: A threshold at the noise scale rejects genuine inliers that happen to sit just outside: 77 of 200 kept, corner error ~1.0 px — looser, not stricter.

Sort into buckets

Some of these hold up and some are the exact mistakes this lesson is built to prevent. Sort them.

Holds up

We want a stable correspondence set between the same skin region photographed weeks apart, then a geometric model that maps Visit 1 to Visit 2 so lesions can be measured.; Phrase the complaint 'SIFT and RANSAC don't match consistently' as four testable hypotheses:; Your MatchQuality already records the right numbers. Read them as a funnel — each row should fall by a sane factor, not collapse.

Breaks

Matches look bad, so loosen everything at once: ratio → 0.9, RANSAC threshold → 10.; RootSIFT the main descriptors, but leave the augmented stable-spot descriptors as raw SIFT, then vstack and match in one FLANN index.

sound

These are stated as this lesson states them — each one survives the edge cases Robust SIFT + RANSAC Landmark Matching (Before/After Registration) puts it through.

flawed

Each of these is lifted from a trap in this deck: reasonable-sounding, and wrong in a way that only shows up once you rely on it.

Predict first

RANSAC is unstable: at threshold 1.0 the corner error is ~1.0 px and only 77/200 true inliers are kept, but at 5.0 px the error drops to ~0.18 px. You switch to USAC_MAGSAC. What is the principled reason MAGSAC's corner error stays ~0.18 px across thresholds 1–8?

Answer it in your own words, now, with nothing to choose from. The options are on the next slide — and picking the right one off a list is an easier skill than producing it.

Correct: It marginalizes the model over a range of noise scales (σ-consensus) and weights every point by its residual, never committing to a single hard inlier/outlier cut.

Why: MAGSAC++ treats σ as unknown and integrates model quality over a σ range, giving each point a soft, residual-decaying weight and solving an iteratively-reweighted least squares. Because nothing is hard-classified at a single τ, the result barely depends on the threshold you pass (it acts as a loose upper bound) — hence the flat ~0.18 px error.

Check your understanding

RANSAC is unstable: at threshold 1.0 the corner error is ~1.0 px and only 77/200 true inliers are kept, but at 5.0 px the error drops to ~0.18 px. You switch to USAC_MAGSAC. What is the principled reason MAGSAC's corner error stays ~0.18 px across thresholds 1–8?

• A. It marginalizes the model over a range of noise scales (σ-consensus) and weights every point by its residual, never committing to a single hard inlier/outlier cut. (correct)
• B. It simply uses a much larger fixed threshold internally, which always includes all the inliers.
• C. It runs more iterations, and with enough iterations any RANSAC variant converges to the same answer.
• D. It discards the threshold and fits ordinary least squares to all correspondences, including the outliers.

Answer: A

Why: MAGSAC++ treats σ as unknown and integrates model quality over a σ range, giving each point a soft, residual-decaying weight and solving an iteratively-reweighted least squares. Because nothing is hard-classified at a single τ, the result barely depends on the threshold you pass (it acts as a loose upper bound) — hence the flat ~0.18 px error.

Why B tempts people

It does not swap in one large fixed cut; a large fixed threshold would admit outliers. MAGSAC stays accurate even at large τ precisely because it down-weights rather than hard-including.

Why C tempts people

Iteration count addresses sampling luck, not the threshold-sensitivity of the consensus criterion; vanilla RANSAC remains τ-sensitive no matter how long it runs.

Why D tempts people

Plain least squares on all 320 points would be wrecked by the 38% outliers; MAGSAC's per-point weighting is exactly what suppresses them.

Discussion prompt

Explain The brief by analogy to something with no Computer Vision in it at all — a queue, a recipe, a map, a bank balance, whatever fits. Then say where your analogy breaks.

Hint: An analogy that never breaks is not an analogy, it is the same idea wearing a hat. Find the seam — that is the part that is actually new.

Comparison matrix

From Milestone 3 — MAGSAC++: every row here is a choice with a cost. Fill the Expected on OpenCV 4.12 column, then say which row you would actually pick and what you give up for it.

Comparison matrix

From Put it together — the end-to-end win: refill the After fixes column from what you know. The rest of the table is as it appeared.

Discussion prompt

Run The accuracy recipe, generalized with this step confiscated:

Robust estimator, soft threshold — MAGSAC++ over fixed-τ RANSAC; pass τ as an upper bound.

Is it still possible? If it is, say what takes its place and what it costs you. If it is not, say exactly what that step was providing that nothing else does.

Hint: A step you can drop for free was never load-bearing. If you cannot drop it, name the thing that goes wrong the moment it is gone.

Discussion prompt

The accuracy recipe, generalized works on the cases you have just seen. Push it to the edge: what is the most degenerate input it still handles — empty, zero, one item, everything equal — and what is the first case where it stops being true? Name the case, not just "it breaks".

Hint: Try the smallest legal input, then the largest, then the one where two things collide. Methods are specified at their edges; the middle takes care of itself.

Eliminate the wrong options

Your fixes raised geometrically-correct matches from ~14% to ~70% and gave a threshold-stable ~0.18 px homography. A reviewer asks which single change most improves the PURITY of the correspondence set fed to RANSAC. Best-supported answer?

3 of these 4 are wrong. Strike them one at a time, and say what rules each one out before you strike the next. The survivor is the answer.

• A. The mutual nearest-neighbor cross-check — it removes asymmetric one-sided matches and drove the largest precision jump (≈13.8% → 59% even before RootSIFT was added).
• B. Switching RANSAC to MAGSAC++, because a better estimator cleans the correspondences before fitting.
• C. CLAHE preprocessing, because local contrast equalization removes all false matches.
• D. Increasing nfeatures, because more keypoints guarantee a higher inlier ratio.

Survives elimination: A

Why: Purity is a property of the correspondence set produced by the matcher. The ablation shows cross-check alone moves correct-match rate from 13.8% to 59% (with RootSIFT compounding it to ~70%). RootSIFT helps description and rescues mole landmarks, but the dominant purity lever is the mutual-NN constraint.

Check your understanding

Your fixes raised geometrically-correct matches from ~14% to ~70% and gave a threshold-stable ~0.18 px homography. A reviewer asks which single change most improves the PURITY of the correspondence set fed to RANSAC. Best-supported answer?

• A. The mutual nearest-neighbor cross-check — it removes asymmetric one-sided matches and drove the largest precision jump (≈13.8% → 59% even before RootSIFT was added). (correct)
• B. Switching RANSAC to MAGSAC++, because a better estimator cleans the correspondences before fitting.
• C. CLAHE preprocessing, because local contrast equalization removes all false matches.
• D. Increasing nfeatures, because more keypoints guarantee a higher inlier ratio.

Answer: A

Why: Purity is a property of the correspondence set produced by the matcher. The ablation shows cross-check alone moves correct-match rate from 13.8% to 59% (with RootSIFT compounding it to ~70%). RootSIFT helps description and rescues mole landmarks, but the dominant purity lever is the mutual-NN constraint.

Why B tempts people

MAGSAC improves how a model is fit from a given set; it does not change which correspondences the matcher emits, so it cannot alter set purity.

Why C tempts people

CLAHE stabilizes detection/description but does not enforce match consistency; the ablation attributes the precision jump to cross-check, not preprocessing.

Why D tempts people

On self-similar skin more keypoints typically add more ambiguous matches, often lowering the inlier ratio rather than raising it — the opposite of a guarantee.

Discussion prompt

You can now stand at a whiteboard and defend the pipeline end to end. Be ready to:

That is stated as though it always holds. Do one of two things: produce a case where it fails, or say precisely what rules such a case out. "It just does" is not on the menu.

Hint: Hunt at the extremes first — zero, one, negative, empty, equal. If every extreme survives, the reason they survive is the proof.

Draw it

One page, no notation unless you need it: draw how these connect — Diagnose Before You Tune · The Descriptor: RootSIFT · The Matcher: Ratio + Mutual NN · The Estimator: RANSAC → MAGSAC++ · Your Turn: Implement All Three. Put an arrow wherever one of them is what makes another possible, and label the arrow with why.