Lesson 27: Numerical Stability

Discussion prompt

Before we open Lesson 27: Numerical Stability: without looking back, what was the main idea of Evaluation Metrics, and what could you do by the end of it that you could not do before?

Hint: One sentence for the idea, one for the skill. If the second one is blank, that is the part to revisit.

Predict first

Let numpy report the exact edges. This block re-imports numpy and prints the four numbers that define the whole lesson - runnable on its own:

Commit before you compute: what does The overflow / underflow thresholds, measured come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: exp(709) is finite, exp(710) is inf

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. The cliff sits at log(max) = 709.78.

Comparison matrix

From The overflow / underflow thresholds, measured: refill the printed value column from what you know. The rest of the table is as it appeared.

Sort into buckets

Every line below is part of the definition of overflow or of underflow — one or the other, never both. Put each where it belongs.

overflow

A result whose magnitude exceeds the largest representable float, so it is rounded to +inf (or -inf).; Once inf enters a computation it usually spreads; inf - inf and inf / inf both give nan.

underflow

A nonzero result whose magnitude is below the smallest representable positive float, rounded to 0.0.; Dangerous when you later divide by it or take its log - 0/0 = nan and log(0) = -inf.

b1

A result whose magnitude exceeds the largest representable float, so it is rounded to +inf (or -inf). Once inf enters a computation it usually spreads: inf - inf and inf / inf both give nan.

b2

A nonzero result whose magnitude is below the smallest representable positive float, rounded to 0.0. Dangerous when you later divide by it or take its log - 0/0 = nan and log(0) = -inf.

Discussion prompt

Overflow is loud - you see inf. Cancellation is silent. Subtract two nearly equal numbers and the leading digits agree and cancel, leaving only the noisy trailing digits.

That is stated as though it always holds. Do one of two things: produce a case where it fails, or say precisely what rules such a case out. "It just does" is not on the menu.

Hint: Hunt at the extremes first — zero, one, negative, empty, equal. If every extreme survives, the reason they survive is the proof.

Discussion prompt

Watch a 1 vanish, then watch log1p rescue log(1 + x) for tiny x. Standalone:

What do you need to know — or decide — before the first line can be written? List everything the problem has to hand you.

Hint: Anything you would have to invent to get started is a thing the problem must supply.

Comparison matrix

From Cancellation, caught in the act: every row here is a choice with a cost. Fill the printed value column, then say which row you would actually pick and what you give up for it.

Discussion prompt

Explain The running example by analogy to something with no Machine Learning in it at all — a queue, a recipe, a map, a bank balance, whatever fits. Then say where your analogy breaks.

Hint: An analogy that never breaks is not an analogy, it is the same idea wearing a hat. Find the seam — that is the part that is actually new.

Discussion prompt

Explain Softmax, and why exp is unavoidable to a student a year behind you. No notation, no jargon they have not met — and it still has to be true.

Hint: If your explanation needs a symbol they have never seen, you are describing the notation rather than the idea.

Predict first

The table runs: np.exp(z) | [inf, inf, inf] · e.sum() | inf

In The naive formula overflows to nan, given the rows so far: what is the next one — the row where step is e / e.sum()?

Correct: e / e.sum() | [nan, nan, nan]

Why: The relationship between the columns, not the individual numbers, is what generates the next row. Each logit exceeds 709.78, so each exp overflows.

Annotate

In The naive formula overflows to nan, before reading the notes: mark where the time actually goes. Which line dominates?

• Each logit exceeds 709.78, so each exp overflows. numpy raises a RuntimeWarning ('overflow encountered in exp') and fills the array with inf - it does not crash, which is exactly why this bug slips into production silently.
• The sum is inf, so every entry becomes inf/inf, which IEEE-754 defines as nan. The probabilities are destroyed even though the true answer is a tame [0.090, 0.245, 0.665].

Put in order

Put the moves of Prove shift invariance in one line into the order they have to happen.

1. Start from softmax(z - c)_i
2. Factor e^{-c} out of each exponential
3. Cancel the common e^{-c}

Why: These are the moves of the worked example in the order it makes them, and each one is set up by the one before it. Subtract c from every logit and write out the definition.

Annotate

From Prove shift invariance in one line — read this one piece at a time. What is each part doing?

On: \( \operatorname{softmax}(z - c)_i = \frac{e^{z_i - c}}{\sum_j e^{z_j - c}} \)

• Subtract c from every logit and write out the definition.
• e^{z_i - c} = e^{z_i} e^{-c}, and likewise in every term of the sum - the same constant factor appears on top and in every summand below.
• The identical nonzero factor e^{-c} in numerator and denominator cancels exactly - no rounding, no approximation. We are left with the original softmax.

Fill in the blanks

From Shift invariance, confirmed numerically — one line has had its right-hand side removed. Put it back.

import numpy as np
z = np.array([1000., 1001., 1002.])
def softmax(x):
e = np.exp(x - x.max()); return e / e.sum()
print(softmax(z).round(6)) # [0.090031 0.244728 0.665241]
print(softmax(np.array([1.,2.,3.])).round(6)) # [0.090031 0.244728 0.665241]
print(np.allclose(softmax(z), softmax(z - 500.))) # True

Why: z is what everything below it consumes, so the wrong expression here fails later and somewhere else. [1000,1001,1002] is [1,2,3] shifted up by 999.

Annotate

Walk the callouts on Shift invariance, confirmed numerically. Each one is a place this is easy to get subtly wrong.

• [1000,1001,1002] is [1,2,3] shifted up by 999. Shift invariance says the softmax is identical - and it is, [0.090031, 0.244728, 0.665241] both ways. Only the gaps (1 and 2) matter, never the absolute scale.
• Any shift works, not just by the max - subtracting 500 gives the same distribution. We choose the max only because it is the shift that guarantees no overflow.

Discussion prompt

Work backwards. The example finished here:

softmax = [0.090031, 0.244728, 0.665241], sums to 1

What was it asked to do, and what must it have been given? Reconstruct the problem from its answer.

Hint: Every quantity in the result had to enter somewhere. Account for each one.

Step through it

Step through Full hand trace of the stable softmax one row at a time. What is driving the change, and what would the row after the last one be?

1. Step 1: i is 0
2. Step 2: i is 1
3. Step 3: i is 2

Predict first

A student writes this, and it looks reasonable:

Softmax is exp(z) / Σ exp(z) - so translate the formula directly, no tricks.

It is wrong. Say what breaks — and say it before you turn the page.

Correct: The formula is mathematically correct but numerically dead: each exp overflows past 709.78, the sum is inf, and inf/inf = nan.

Subtract the max first - shift invariance makes it mathematically identical and numerically safe.

Why: The formula is mathematically correct but numerically dead: each exp overflows past 709.78, the sum is inf, and inf/inf = nan. The model's predictions become nan and every downstream loss and gradient turns to nan too.

Discussion prompt

The rule this trap just fixed:

Subtract the max first - shift invariance makes it mathematically identical and numerically safe.

Now break it on purpose. Build a case that violates it and follow the consequences until something visibly fails. Where does the failure first show up — and would you have noticed it if you had not been looking?

Hint: The dangerous rules are the ones whose violation still produces an answer. If yours fails loudly, try to find one that fails quietly.

Put in order

Put the moves of Derive the log-sum-exp identity into the order they have to happen.

1. Factor e^{z*} out of the sum
2. Take the log; log of a product is a sum of logs
3. Every shifted exponent is <= 0, so the inner sum is safe

Why: These are the moves of the worked example in the order it makes them, and each one is set up by the one before it. Let z* = max z. Every term e^{z_j} = e^{z} e^{z_j - z}, so the common factor e^{z*} pulls out of the whole sum.

Draw it

Translate both ways. First write the expression above as a sentence with no symbols in it at all. Then cover it, and write your sentence back as notation. If the two versions disagree, the disagreement is the thing to fix.

Sort into buckets

These are the pieces of Lesson 27: Numerical Stability, out of order. Put each one back under the part of the lesson it belongs to.

How floats break

A float has finite range; Where exp overflows; The overflow / underflow thresholds, measured

Softmax that survives

The running example; Softmax, and why exp is unavoidable; The naive formula overflows to nan

The log-sum-exp trick

Why we need log Σ exp; Derive the log-sum-exp identity; LSE by hand on z = [1000, 1001, 1002]

s1

How floats break is where Lesson 27: Numerical Stability puts A float has finite range, Where exp overflows, The overflow / underflow thresholds, measured. Knowing which part of the lesson a problem belongs to is most of knowing which method to reach for.

s2

Softmax that survives is where Lesson 27: Numerical Stability puts The running example, Softmax, and why exp is unavoidable, The naive formula overflows to nan. Knowing which part of the lesson a problem belongs to is most of knowing which method to reach for.

s3

The log-sum-exp trick is where Lesson 27: Numerical Stability puts Why we need log Σ exp, Derive the log-sum-exp identity, LSE by hand on z = [1000, 1001, 1002]. Knowing which part of the lesson a problem belongs to is most of knowing which method to reach for.

Fill in the blanks

Naive LSE vs stable LSE vs scipy, with the scaffolding fading: two lines are gone now — fill both.

import numpy as np
from scipy.special import logsumexp
z = np.array([1000., 1001., 1002.])
m = z.max()
naive = np.log(np.sum(np.exp(z))) # inf
stable = m + np.log(np.sum(np.exp(z - m)))
print(naive) # inf
print(round(stable, 6)) # 1002.407606
print(round(float(logsumexp(z)), 6)) # 1002.407606

Why: Reproducing these unaided, rather than reading them, is what tells you the method has transferred. The naive log(sum(exp(z))) overflows inside the exp and returns inf.

Predict first

Compute log p and the loss for true class 2, all through LSE - never a raw softmax. Standalone:

Commit before you compute: what does Stable log-softmax and cross-entropy come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: CE = -log p_2 = 0.407606

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. The true class is 2, whose log-probability is -0.407606, so the loss is 0.407606.

Draw it

Draw what Stable log-softmax and cross-entropy just did — the shape of it, not the line-by-line working. One picture, labels only where you need them. Then check it against the steps: anything you could not draw is a step you followed rather than understood.

Fill in the blanks

From Underflow: the same fix rescues tiny logits — one line has had its right-hand side removed. Put it back.

import numpy as np
z = np.array([-1000., -1001., -1002.])
e = np.exp(z)
print(e) # [0. 0. 0.]
print(e / e.sum()) # [nan nan nan] 0/0
def softmax(x):
e = np.exp(x - x.max()); return e / e.sum()
print(softmax(z).round(6)) # [0.665241 0.244728 0.090031]

Why: e is what everything below it consumes, so the wrong expression here fails later and somewhere else. Every logit is below -745, so each exp rounds to exactly 0.0; the denominator is 0, and 0/0 is nan.

Fill in the blanks

Forming XᵀX squares the condition number, with the scaffolding fading: two lines are gone now — fill both.

import numpy as np
X = np.array([[1., 1.0000],
[1., 1.0001],
[1., 1.0002]])
kX = np.linalg.cond(X)
kG = np.linalg.cond(X.T @ X)
print(f'___') # 2.4497e+04 cond(X)
print(f'___') # 6.0012e+08 cond(XtX)
print(f'___') # 6.0012e+08 = cond(X)^2

Why: Reproducing these unaided, rather than reading them, is what tells you the method has transferred. Singular values of XtX are the squares of X's singular values, so kappa(XtX) = kappa(X)^2.

Comparison matrix

From Forming XᵀX squares the condition number: refill the value (verified) column from what you know. The rest of the table is as it appeared.

Fill in the blanks

From The Hilbert matrix: conditioning explodes — one line has had its right-hand side removed. Put it back.

import numpy as np
from scipy.linalg import hilbert
H = hilbert(8)
print(f'np.linalg.solve(H, b)') # 1.5258e+10
b = H @ np.ones(8) # exact rhs, true x = ones
xs = ___
xi = np.linalg.inv(H) @ b
print(f'___') # 1.3089e-07 solve
print(f'___') # 9.5367e-07 inv (worse)

Why: xs is what everything below it consumes, so the wrong expression here fails later and somewhere else. log10(1.53e10) = 10.18. Of double precision's ~16 digits, about 10 are eaten by conditioning, leaving ~6.

Draw it

Draw what The Hilbert matrix: conditioning explodes just did — the shape of it, not the line-by-line working. One picture, labels only where you need them. Then check it against the steps: anything you could not draw is a step you followed rather than understood.

Step through it

Step through How bad it gets with size one row at a time. What is driving the change, and what would the row after the last one be?

1. Step 1: n is 4
2. Step 2: n is 6
3. Step 3: n is 8
4. Step 4: n is 10

Predict first

A student writes this, and it looks reasonable:

To solve Ax = b, the math says x = A⁻¹ b - so compute the inverse and multiply.

It is wrong. Say what breaks — and say it before you turn the page.

Correct: Explicitly forming A^{-1} runs a full extra pass of arithmetic on the already ill-conditioned matrix, amplifying round-off.

Never form the inverse. Factor and solve directly - and if κ is huge, regularize.

Why: Explicitly forming A^{-1} runs a full extra pass of arithmetic on the already ill-conditioned matrix, amplifying round-off. On Hilbert(8) the recovered x is off by 9.5e-7 - about 7x the error of a direct solve.

Sort into buckets

Some of these hold up and some are the exact mistakes this lesson is built to prevent. Sort them.

Holds up

Add 1 to 10¹⁶ and the 1 falls off the end of the 16 available digits. Subtract 10¹⁶ back and you get 0 - the 1 is gone forever. The relative error just exploded.; One dataset for the whole lesson: a classifier emits three logits (pre-softmax scores) for three classes. They happen to be large.; Softmax turns logits into a probability vector - all entries positive and summing to 1 - by exponentiating and normalizing:

Breaks

Softmax is exp(z) / Σ exp(z) - so translate the formula directly, no tricks.; To solve Ax = b, the math says x = A⁻¹ b - so compute the inverse and multiply.

sound

These are stated as this lesson states them — each one survives the edge cases Lesson 27: Numerical Stability puts it through.

flawed

Each of these is lifted from a trap in this deck: reasonable-sounding, and wrong in a way that only shows up once you rely on it.

Discussion prompt

Explain A floor under the smallest direction to a student a year behind you. No notation, no jargon they have not met — and it still has to be true.

Hint: If your explanation needs a symbol they have never seen, you are describing the notation rather than the idea.

Discussion prompt

Explain Add lambda I to the matrix by analogy to something with no Machine Learning in it at all — a queue, a recipe, a map, a bank balance, whatever fits. Then say where your analogy breaks.

Hint: An analogy that never breaks is not an analogy, it is the same idea wearing a hat. Find the seam — that is the part that is actually new.

Put in order

Put the moves of lambda I shifts every eigenvalue up by lambda into the order they have to happen.

1. Start from an eigenpair of A
2. Apply (A + lambda I) to the same v
3. Same eigenvector, eigenvalue mu + lambda

Why: These are the moves of the worked example in the order it makes them, and each one is set up by the one before it. Let v be an eigenvector of symmetric A with eigenvalue mu: Av = mu v.

Annotate

From lambda I shifts every eigenvalue up by lambda — read this one piece at a time. What is each part doing?

On: \( \kappa(A + \lambda I) = \frac{\mu_{\max} + \lambda}{\mu_{\min} + \lambda} \;\ll\; \frac{\mu_{\max}}{\mu_{\min}} = \kappa(A) \)

• Let v be an eigenvector of symmetric A with eigenvalue mu: Av = mu v.
• (A + lambda I)v = Av + lambda v = mu v + lambda v, using Iv = v.
• So A and A + lambda I share every eigenvector, and each eigenvalue rises by exactly lambda. The smallest eigenvalue mu_min - the one that made kappa huge - becomes mu_min + lambda, lifted off the floor.

Predict first

The table runs: cond | 1.5258e+10 | 1.6969e+03 · min eigenvalue | 1.1115e-10 | 1.0000e-03

In Ridge on the Hilbert matrix, given the rows so far: what is the next one — the row where quantity is max eigenvalue?

Correct: max eigenvalue | 1.6959 | 1.6969

Why: The relationship between the columns, not the individual numbers, is what generates the next row. Adding lambda I lifts the smallest eigenvalue from 1.11e-10 to almost exactly 1e-3 (= lambda, since lambda swamps the tiny original), while the largest (~1.70) barely moves.

Step through it

Step through Ridge on the Hilbert matrix one row at a time. What is driving the change, and what would the row after the last one be?

1. Step 1: quantity is cond
2. Step 2: quantity is min eigenvalue
3. Step 3: quantity is max eigenvalue

Discussion prompt

Logits of just ~100 - trivial for float64 - already overflow in float32. The max-subtraction rescues both. Standalone:

What do you need to know — or decide — before the first line can be written? List everything the problem has to hand you.

Hint: Anything you would have to invent to get started is a thing the problem must supply.

Discussion prompt

Work backwards. The example finished here:

stable softmax works in float32 too

What was it asked to do, and what must it have been given? Reconstruct the problem from its answer.

Hint: Every quantity in the result had to enter somewhere. Account for each one.

Predict first

PyTorch's softmax, logsumexp, and cross_entropy all do the max trick internally, so they match our hand-derived values on the same z. Standalone:

Commit before you compute: what does torch computes the same stable numbers come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: torch cross_entropy(target=2) = 0.407606

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. cross_entropy takes RAW logits and internally computes LSE - z_true, matching our hand value 0.407606.

Annotate

Walk the callouts on torch computes the same stable numbers. Each one is a place this is easy to get subtly wrong.

• Identical to our numpy stable softmax - torch subtracts the max under the hood, so the same z that overflows a naive formula gives a clean distribution.
• cross_entropy takes RAW logits and internally computes LSE - z_true, matching our hand value 0.407606. This is why you pass logits, never softmax outputs, to the loss - double-softmaxing is a classic bug.

Put in order

These are the steps of The numerical-stability toolkit, scrambled. Put them back in order before the next slide shows you.

1. Softmax / log-softmax: subtract max(z) before exp - shift-invariant, so the answer is unchanged and nothing overflows
2. Sums of exps (LSE): use LSE(z) = z* + log Σ e^(zᵢ - z*); feed raw logits to cross-entropy, never softmax outputs
3. Near-equal subtractions: avoid cancellation with log1p / expm1; do not form the tiny difference
4. Linear solves: np.linalg.solve / lstsq, never inv; check κ and expect to lose ~log₁₀ κ digits
5. Ill-conditioned (κ huge): regularize with A + λI - it lifts the smallest eigenvalue and shrinks κ
6. Low precision: float32 overflows exp at ~88.7; use mixed precision (FP16 compute + FP32 master state, loss scaling)

Why: This is the order the recipe itself gives. Recalling the sequence without the slide in front of you is the difference between recognising the method and being able to run it — most of what goes wrong in practice is a step done out of turn.

Discussion prompt

The numerical-stability toolkit works on the cases you have just seen. Push it to the edge: what is the most degenerate input it still handles — empty, zero, one item, everything equal — and what is the first case where it stops being true? Name the case, not just "it breaks".

Hint: Try the smallest legal input, then the largest, then the one where two things collide. Methods are specified at their edges; the middle takes care of itself.

Eliminate the wrong options

Subtracting max(z) before exponentiating in softmax is valid because:

3 of these 4 are wrong. Strike them one at a time, and say what rules each one out before you strike the next. The survivor is the answer.

• A. softmax is shift-invariant: softmax(z) = softmax(z - c) for any constant c
• B. it makes the output more accurate than the true softmax
• C. the maximum logit is always negligible
• D. it rescales the logits to unit variance

Survives elimination: A

Why: e^{-c} factors out of the numerator and every term of the denominator and cancels exactly, so softmax(z - c) = softmax(z). Choosing c = max makes every exponent <= 0 (each exp <= 1), so nothing overflows - with zero change to the result.

Check your understanding

Subtracting max(z) before exponentiating in softmax is valid because:

• A. softmax is shift-invariant: softmax(z) = softmax(z - c) for any constant c (correct)
• B. it makes the output more accurate than the true softmax
• C. the maximum logit is always negligible
• D. it rescales the logits to unit variance

Answer: A

Why: e^{-c} factors out of the numerator and every term of the denominator and cancels exactly, so softmax(z - c) = softmax(z). Choosing c = max makes every exponent <= 0 (each exp <= 1), so nothing overflows - with zero change to the result.

Why B tempts people

The result is mathematically identical, not more accurate - the point is to avoid overflow (inf/inf = nan), not to change the answer. The exact-cancellation proof shows the value is unchanged.

Why C tempts people

The max can be enormous (1002 here); it is the opposite of negligible. Subtracting it is exactly what pulls the exponents down to a safe range.

Why D tempts people

Subtracting a constant is a shift, not a rescale; it does not touch the variance. Rescaling to unit variance would divide the logits and change the softmax output.

Predict first

The log-sum-exp trick rewrites log Σ e^{zᵢ} as:

Answer it in your own words, now, with nothing to choose from. The options are on the next slide — and picking the right one off a list is an easier skill than producing it.

Correct: z* + log Σ e^{zᵢ - z}, where z = max zᵢ

Why: Factor e^{z} out of the sum, then take the log: log(e^{z} Σ e^{zᵢ - z}) = z + log Σ e^{zᵢ - z*}. The shifted exponents are all <= 0, so the inner sum stays in (0, n] - no overflow - and the identity is exact.

Check your understanding

The log-sum-exp trick rewrites log Σ e^{zᵢ} as:

• A. z* + log Σ e^{zᵢ - z}, where z = max zᵢ (correct)
• B. Σ log(zᵢ)
• C. log(z*) + Σ e^{zᵢ}
• D. max over log(zᵢ)

Answer: A

Why: Factor e^{z} out of the sum, then take the log: log(e^{z} Σ e^{zᵢ - z}) = z + log Σ e^{zᵢ - z*}. The shifted exponents are all <= 0, so the inner sum stays in (0, n] - no overflow - and the identity is exact.

Why B tempts people

Σ log(zᵢ) is log of a product of the logits - a completely different quantity, and it takes log of the raw logits rather than of a sum of exponentials.

Why C tempts people

This keeps a raw Σ e^{zᵢ} (which still overflows) and wrongly takes log of z. The max is added OUTSIDE the log as z, not as log(z*), and the surviving sum must be the shifted one.

Why D tempts people

The trick preserves the full sum of exponentials; it does not collapse to a max over logs. That would discard all but one term.

Predict first

A linear system has condition number kappa ~ 10^10. Solving it in double precision (~16 digits) leaves roughly:

Answer it in your own words, now, with nothing to choose from. The options are on the next slide — and picking the right one off a list is an easier skill than producing it.

Correct: ~6 reliable digits - you lose about log10(kappa) ~ 10

Why: Each factor of 10 in kappa costs about one digit, so kappa ~ 10^10 loses ~10 of the ~16 available, leaving ~6 reliable digits. That is exactly what the Hilbert(8) solve showed: cond 1.5e10, error ~1.3e-7 (wrong near the 7th digit).

Check your understanding

A linear system has condition number kappa ~ 10^10. Solving it in double precision (~16 digits) leaves roughly:

• A. ~6 reliable digits - you lose about log10(kappa) ~ 10 (correct)
• B. all 16 digits - kappa does not affect accuracy
• C. 0 digits - the answer is completely meaningless
• D. exactly 10 extra digits are gained

Answer: A

Why: Each factor of 10 in kappa costs about one digit, so kappa ~ 10^10 loses ~10 of the ~16 available, leaving ~6 reliable digits. That is exactly what the Hilbert(8) solve showed: cond 1.5e10, error ~1.3e-7 (wrong near the 7th digit).

Why B tempts people

Conditioning directly bounds the achievable accuracy; no algorithm can dodge it. A large kappa erodes digits regardless of whether you use solve, inv, or lstsq.

Why C tempts people

You lose ~10 digits, not all 16 - about 6 remain, so the answer is degraded but still meaningful. It becomes meaningless only when log10(kappa) approaches the ~16 available (around n=12 for Hilbert).

Why D tempts people

A large kappa LOSES accuracy, it does not gain it. The sign of the effect is backwards - conditioning is a cost, never a bonus.

Eliminate the wrong options

Why prefer np.linalg.solve(A, b) over np.linalg.inv(A) @ b for an ill-conditioned A?

3 of these 4 are wrong. Strike them one at a time, and say what rules each one out before you strike the next. The survivor is the answer.

• A. Forming the explicit inverse does more arithmetic on the ill-conditioned matrix, amplifying round-off - solve factors once and is more accurate
• B. inv returns a mathematically different, wrong answer
• C. solve automatically adds ridge regularization
• D. inv only works on symmetric matrices

Survives elimination: A

Why: solve does one stable LU factorization plus back-substitution; inv computes the full inverse (extra passes over the bad matrix) and then multiplies. On Hilbert(8) solve's error was 1.3e-7 versus inv's 9.5e-7 - same math, worse numerics. Better still is lstsq (QR/SVD), which never forms A explicitly.

Check your understanding

Why prefer np.linalg.solve(A, b) over np.linalg.inv(A) @ b for an ill-conditioned A?

• A. Forming the explicit inverse does more arithmetic on the ill-conditioned matrix, amplifying round-off - solve factors once and is more accurate (correct)
• B. inv returns a mathematically different, wrong answer
• C. solve automatically adds ridge regularization
• D. inv only works on symmetric matrices

Answer: A

Why: solve does one stable LU factorization plus back-substitution; inv computes the full inverse (extra passes over the bad matrix) and then multiplies. On Hilbert(8) solve's error was 1.3e-7 versus inv's 9.5e-7 - same math, worse numerics. Better still is lstsq (QR/SVD), which never forms A explicitly.

Why B tempts people

In exact arithmetic both give the same x; the difference is floating-point round-off and speed, not the underlying algebra.

Why C tempts people

solve does no regularization - it factors whatever matrix you hand it. Ridge is the separate A + lambda I move.

Why D tempts people

inv works on any square non-singular matrix, symmetric or not. Symmetry is irrelevant to the reason to avoid it.

Discussion prompt

Build rules: type every line yourself, run after each line, and test on [1000, 1001, 1002] so the naive version visibly dies. When something prints nan, read why - do not paper over it.

That is stated as though it always holds. Do one of two things: produce a case where it fails, or say precisely what rules such a case out. "It just does" is not on the menu.

Hint: Hunt at the extremes first — zero, one, negative, empty, equal. If every extreme survives, the reason they survive is the proof.

Comparison matrix

From Milestone 3 - log-sum-exp vs scipy: every row here is a choice with a cost. Fill the log Σ exp z column, then say which row you would actually pick and what you give up for it.

Comparison matrix

From The full program: refill the value column from what you know. The rest of the table is as it appeared.

Draw it

One page, no notation unless you need it: draw how these connect — How floats break · Softmax that survives · The log-sum-exp trick · Conditioning of a solve · Ridge shrinks kappa · Precision & scale. Put an arrow wherever one of them is what makes another possible, and label the arrow with why.