L12 · Exploit Mitigations Overview: NX, Stack Canaries, ASLR

CS 161, Lesson 12, in 50 slides and code mode. It explains why mitigations exist at all when you are stuck writing C, then covers non-executable pages - NX, W^X, and DEP - which stop injected shellcode; stack canaries, which detect a contiguous overwrite of the saved rip; and ASLR, which randomizes absolute addresses but not relative offsets. It gives the consolidated table mapping each mitigation to its attack and its bypass, and explains why combining all three is synergistic. This is an overview only, since the deep bypasses come in Week 5. The examples are toys running in a sandbox, and it is anchored to textbook sections 4.4, 4.5, 4.8, and 4.11.

Subject: Computer Security · 86 slides · code lesson

Open the interactive version of this deck · Homework for this lesson

What this lesson covers

The lesson, slide by slide

1. Raising the Bar

Title

CS 161 · Lesson 12 of 45

exploit mitigations · NX / W^X · stack canaries · ASLR · defense-in-depth · the Week 5 preview

2. By the end of this lesson you can…

Objectives

  1. Explain why mitigations exist even though they don't make C memory-safe, and name the only real cure.
  2. Describe NX / W^X / DEP (write XOR execute) and the exact stack-smash it makes the CPU fault on.
  3. Place a stack canary in the frame and explain why a contiguous overflow of the rip trips it.
  4. Describe ASLR, the page-boundary entropy limit, and why it randomizes absolute addresses but not relative offsets.
  5. Fill in the mitigation → attack stopped → how bypassed table and argue why combining all three is synergistic.

3. What survived from L11 · Memory-Safe Languages, Safer C, and the Secure…?

Warm-up

Discussion prompt

Before we open L12 · Exploit Mitigations Overview: NX, Stack Canaries, ASLR: without looking back, what was the main idea of L11 · Memory-Safe Languages, Safer C, and the Secure Software Process, and what could you do by the end of it that you could not do before?

Hint: One sentence for the idea, one for the skill. If the second one is blank, that is the part to revisit.

Answer:

CS 161, Lesson 11, in 50 slides and code mode. It lays out the spectrum of memory-safety fixes: memory-safe languages, which are the only 100% guarantee; Rust's ownership and borrow model, which achieves that without a garbage collector; safer C through pre- and post-conditions and bounds-checked library calls, replacing gets with fgets, strcpy with strncpy or strlcpy, and sprintf with snprintf; and the secure-development process, covering controlled crashes, code review, fuzzing, Valgrind, sanitizers, static analyzers, and the SDL. It is anchored to textbook sections 4.1 to 4.3.

4. A note before we start: this is defensive

Concept

Every example here is a toy in a sandbox, for authorized security education — exactly how the textbook teaches it. We study the mitigations so we can deploy and reason about them.

Shellcode stays abstract as [shellcode]; we never write real attack bytes. The skill being built is knowing which defense stops which step of the L8 attack.

5. Break it if you can: A note before we start: this is defensive

Counterexample

Discussion prompt

Every example here is a toy in a sandbox, for authorized security education — exactly how the textbook teaches it. We study the mitigations so we can deploy and reason about them.

That is stated as though it always holds. Do one of two things: produce a case where it fails, or say precisely what rules such a case out. "It just does" is not on the menu.

Hint: Hunt at the extremes first — zero, one, negative, empty, equal. If every extreme survives, the reason they survive is the proof.

Answer:

Shellcode stays abstract as [shellcode]; we never write real attack bytes. The skill being built is knowing which defense stops which step of the L8 attack.

6. Why Mitigations Exist

Section

Part 1 · §4.4

7. Sometimes you're stuck in C

Concept

The clean fix from L8 was: use a memory-safe language. But in the real world you often can't — you inherit a huge legacy C codebase (a kernel, a daemon, firmware) that you cannot fully audit or rewrite this quarter.

mitigation — A code-hardening defense that makes a known class of exploit HARDER to pull off — typically by making a failed attempt CRASH instead of succeed.

Mitigations assume the bug is still there and try to stop the exploitation anyway.

8. By analogy: Sometimes you're stuck in C

Analogy

Discussion prompt

Explain Sometimes you're stuck in C by analogy to something with no Computer Security in it at all — a queue, a recipe, a map, a bank balance, whatever fits. Then say where your analogy breaks.

Hint: An analogy that never breaks is not an analogy, it is the same idea wearing a hat. Find the seam — that is the part that is actually new.

Answer:

Mitigations assume the bug is still there and try to stop the exploitation anyway.

9. Mitigations are not a cure

Concept

A mitigation raises the bar; it does not remove the vulnerability. The book is blunt about the limit.

the thesis — The only way to prevent all memory safety exploits is to use a memory-safe language.

Mitigations DOMitigations DON'T
make common exploits harderremove the underlying bug
turn many exploits into crashesmake C memory-safe
buy defense-in-depthstop a determined, well-resourced attacker forever

10. Fill in: Mitigations DON'T for Mitigations are not a cure

Comparison

Comparison matrix

From Mitigations are not a cure: refill the Mitigations DON'T column from what you know. The rest of the table is as it appeared.

Mitigations DOMitigations DON'T
make common exploits harderremove the underlying bug
turn many exploits into crashesmake C memory-safe
buy defense-in-depthstop a determined, well-resourced attacker forever

11. Defense-in-depth and the arms race

Intuition

No single mitigation is foolproof, so we stack them — like a castle with a moat AND walls AND guards. If one is bypassed, the next still stands. That layering is defense-in-depth.

Each mitigation provokes a new bypass, which provokes a new mitigation: an ongoing arms race. We're previewing one round of it; Week 5 is the rematch.

12. Teach it back: Defense-in-depth and the arms race

Explain it

Discussion prompt

Explain Defense-in-depth and the arms race to a student a year behind you. No notation, no jargon they have not met — and it still has to be true.

Hint: If your explanation needs a symbol they have never seen, you are describing the notation rather than the idea.

Answer:

No single mitigation is foolproof, so we stack them — like a castle with a moat AND walls AND guards. If one is bypassed, the next still stands. That layering is defense-in-depth.

13. Why stacking is more than the sum

Intuition

Here's the key payoff. Each mitigation forces the attacker to solve a different problem. NX forces them to reuse existing code. A canary forces them to know a secret value. ASLR forces them to know secret addresses.

Combined, the attacker must defeat all of them at once — usually meaning they must find multiple, distinct vulnerabilities (e.g. one to leak memory, one to write). That multiplication of difficulty is what we mean by synergistic.

14. What has to happen first: Recall: the L8 attack chain we're defending against

Ranking

Put in order

Put the moves of Recall: the L8 attack chain we're defending against into the order they have to happen.

  1. Link 1 — the code is in a writable buffer
  2. Link 2 — the overwrite is one contiguous run
  3. Link 3 — the attacker hardcodes &buf

Why: These are the moves of the worked example in the order it makes them, and each one is set up by the one before it. The shellcode sits in buf, a region the program writes to.

15. Recall: the L8 attack chain we're defending against

Worked example

Every mitigation today targets one link of the classic stack smash from L8. Keep this chain on screen.

// classic stack smash (L8), char buf[8]:
payload = [shellcode (8B)]   // 1. attacker-supplied code
        + [4 garbage bytes]  // 2. spacer over the sfp
        + [address of buf]   // 3. overwrite rip (contiguous!)
// on return: eip = &buf -> CPU EXECUTES the buffer bytes

Link 1 — the code is in a writable buffer

Why: The shellcode sits in buf, a region the program writes to. NX will attack this assumption.

Link 2 — the overwrite is one contiguous run

Why: Bytes climb from buf, across the sfp, into the rip with no gaps. The stack canary will attack this.

Link 3 — the attacker hardcodes &buf

Why: The payload contains a fixed absolute address. ASLR will attack this.

Attack linkAssumptionMitigation that breaks it
execute injected codewritable memory is executableNX / W^X
contiguous rip overwriteno guard between buf and ripstack canary
hardcoded &bufaddresses are predictableASLR

16. What each one costs: Recall: the L8 attack chain we're defending…

Trade off

Comparison matrix

From Recall: the L8 attack chain we're defending against: every row here is a choice with a cost. Fill the Assumption column, then say which row you would actually pick and what you give up for it.

Attack linkAssumptionMitigation that breaks it
execute injected codewritable memory is executableNX / W^X
contiguous rip overwriteno guard between buf and ripstack canary
hardcoded &bufaddresses are predictableASLR

17. Something is wrong here: 'mitigations make C memory-safe'

Anomaly

Predict first

A student writes this, and it looks reasonable:

We turned on NX, canaries, and ASLR. Is the program now memory-safe?

It is wrong. Say what breaks — and say it before you turn the page.

Correct: Tempting — but the buffer overflow bug is still in the code.

We turned on NX, canaries, and ASLR. Is the program now memory-safe?

Why: Tempting — but the buffer overflow bug is still in the code. Mitigations only make this particular exploit harder; a strong-enough attacker (or a new bug) gets through.

18. Trap: 'mitigations make C memory-safe'

Trap

The trap

We turned on NX, canaries, and ASLR. Is the program now memory-safe?

Conclude the C program is now memory-safe

Why: Tempting — but the buffer overflow bug is still in the code. Mitigations only make this particular exploit harder; a strong-enough attacker (or a new bug) gets through.

The fix

We turned on NX, canaries, and ASLR. Is the program now memory-safe?

It is only HARDER to exploit, not safe

Why: §4.4: the only way to prevent ALL memory-safety exploits is a memory-safe language. Mitigations are defense-in-depth — they raise the bar, they don't erase the vulnerability.

19. Non-Executable Pages

Section

Part 2 · §4.5

20. Code or data — pick one, never both

Intuition

The injected-shellcode attack relies on one quiet assumption: the bytes you wrote into a buffer can later be run as instructions. NX simply forbids that combination.

Think of every memory page as a room with a sign: it's either a writing room (you can change what's inside) or a performance stage (the CPU can execute it) — never both at once.

21. Write XOR Execute

Concept

non-executable pages (NX) — Each memory page is set writable XOR executable, never both. If a page is writable, the CPU refuses to execute its bytes as instructions; if executable, it can't be written.

Another way to say the same rule: don't allow eip to ever hold the address of a non-executable page. The instant ret would jump into a writable page, the CPU faults.

NameStands forWhere you'll see it
W^XWrite XOR ExecuteOpenBSD, conceptual name
DEPData Execution PreventionWindows
NX bitNo-eXecute page-table bitx86 hardware / Linux

22. Take the definitions apart: mitigation vs non-executable pages…

Definition probe

Sort into buckets

Every line below is part of the definition of mitigation or of non-executable pages (NX) — one or the other, never both. Put each where it belongs.

mitigation
A code-hardening defense that makes a known class of exploit HARDER to pull off; typically by making a failed attempt CRASH instead of succeed.
non-executable pages (NX)
Each memory page is set writable XOR executable, never both.; If a page is writable, the CPU refuses to execute its bytes as instructions; if executable, it can't be written.
b1
A code-hardening defense that makes a known class of exploit HARDER to pull off — typically by making a failed attempt CRASH instead of succeed.
b2
Each memory page is set writable XOR executable, never both. If a page is writable, the CPU refuses to execute its bytes as instructions; if executable, it can't be written.

23. Where the rule comes from: pages, not bytes

Intuition

The permission lives on the page, not on individual bytes. Memory is carved into pages (typically 4096 bytes), and each page carries permission bits the hardware enforces: read, write, execute.

W^X is just a policy on those bits: for every page, the execute bit and the write bit are never both set. The stack and heap get write-but-not-execute; the code section gets execute-but-not-write.

24. Plan first: Why the L8 shellcode injection now faults

Step zero

Discussion prompt

Why the L8 shellcode injection now faults — before any calculation: what is the plan? Name the moves in order, in plain English, without doing the arithmetic.

Hint: It starts with: The overflow still succeeds — rip is overwritten

Answer:

  1. The overflow still succeeds — rip is overwritten
  2. ret loads &buf into eip
  3. The CPU faults instead of executing

25. Why the L8 shellcode injection now faults

Worked example

Re-run the L8 'inject into buf' attack (Strategy 2) on a system with NX enabled, and watch exactly where it dies.

payload = [shellcode (8B)]   // buf[0..7]  -- on a WRITABLE stack page
        + [4 garbage bytes]  // sfp
        + [address of buf]   // rip = &buf
// ret: eip <- &buf  -->  &buf is on a writable (==non-executable) page

The overflow still succeeds — rip is overwritten

Why: NX does nothing to stop the write. The buffer overflows and the rip becomes &buf exactly as before.

ret loads &buf into eip

Why: The CPU is now about to fetch its next instruction from the stack page that holds buf.

The CPU faults instead of executing

Why: buf lives on a writable stack page, so under W^X that page is non-executable. Fetching an instruction from it raises a fault and the program CRASHES — the exploit fails.

StepWithout NXWith NX
overflow buf, overwrite ripsucceedssucceeds
ret sets eip = &bufsucceedssucceeds
execute bytes at bufruns shellcodeFAULT — crash

26. Inspect it line by line: Why the L8 shellcode injection now faults

Error analysis

Annotate

Walk the callouts on Why the L8 shellcode injection now faults. Each one is a place this is easy to get subtly wrong.

  • NX does nothing to stop the write. The buffer overflows and the rip becomes &buf exactly as before.
  • The CPU is now about to fetch its next instruction from the stack page that holds buf.
  • buf lives on a writable stack page, so under W^X that page is non-executable. Fetching an instruction from it raises a fault and the program CRASHES — the exploit fails.

27. Preview of the NX bypass: reuse, don't inject

Concept

NX stops you from running code you placed in a writable buffer. It does nothing about code that is already loaded and already executable — the program's own functions, and the C library.

So the attacker stops injecting and starts reusing: point the rip at system() in libc (return-to-libc, L13) or chain together existing instruction fragments (ROP, L14). The deep mechanics are Week 5 — for now, just know NX is defeated by code reuse.

28. Guess the shape of the answer: Reading a process's page permissions

Estimation

Predict first

Under W^X, every region a process maps is either writable or executable. Walk the typical layout and decide which exploit step each region blocks.

Commit before you compute: what does Reading a process's page permissions come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: Code and libc are r-x (not writable)

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. These are executable, so the rip CAN usefully point here — which is exactly the door code-reuse attacks walk through.

29. Reading a process's page permissions

Worked example

Under W^X, every region a process maps is either writable or executable. Walk the typical layout and decide which exploit step each region blocks.

// simplified /proc/<pid>/maps under W^X (perms column):
//   code (.text)   r-x   executable, NOT writable
//   data/.bss      rw-   writable,   NOT executable
//   heap           rw-   writable,   NOT executable
//   stack          rw-   writable,   NOT executable
//   libc .text     r-x   executable, NOT writable

Stack and heap are rw- (not executable)

Why: Any shellcode the attacker writes into a buffer lands here, so the CPU faults the moment eip points at it — injection is dead.

Code and libc are r-x (not writable)

Why: These are executable, so the rip CAN usefully point here — which is exactly the door code-reuse attacks walk through.

RegionPermsInject here?Reuse here?
stack / heaprw-blocked (NX faults)n/a — not executable
program .textr-xblocked (not writable)yes — ret2existing-code
libc .textr-xblocked (not writable)yes — ret2libc

30. Work backwards from the answer: Reading a process's page permissions

Reverse engineer

Discussion prompt

Work backwards. The example finished here:

Code and libc are r-x (not writable)

What was it asked to do, and what must it have been given? Reconstruct the problem from its answer.

Hint: Every quantity in the result had to enter somewhere. Account for each one.

Answer:

Under W^X, every region a process maps is either writable or executable. Walk the typical layout and decide which exploit step each region blocks.

31. Something is wrong here: 'NX stops all control-flow hijacking'

Anomaly

Predict first

A student writes this, and it looks reasonable:

With NX on, can the attacker still hijack control flow?

It is wrong. Say what breaks — and say it before you turn the page.

Correct: NX only blocks executing injected bytes on writable pages.

With NX on, can the attacker still hijack control flow?

Why: NX only blocks executing injected bytes on writable pages. The attacker can still overwrite the rip and jump — they just have to aim at code that's already executable.

32. Trap: 'NX stops all control-flow hijacking'

Trap

The trap

With NX on, can the attacker still hijack control flow?

Conclude NX makes control-flow hijacking impossible

Why: NX only blocks executing injected bytes on writable pages. The attacker can still overwrite the rip and jump — they just have to aim at code that's already executable.

The fix

With NX on, can the attacker still hijack control flow?

NX stops INJECTION; code reuse defeats it

Why: §4.5: ret2libc / ROP redirect to existing executable code (libc, the binary). The rip overwrite still works — NX only changes where the rip can usefully point.

33. Stack Canaries

Section

Part 3 · §4.8

34. The canary in the coal mine

Intuition

Miners once carried a canary underground: the bird is more sensitive to toxic gas, so if it died, the miners knew to flee before the danger reached them. The bird is sacrificial — an early warning.

A stack canary is the same idea in software: a sacrificial value placed in the line of fire. If a contiguous overflow corrupts it, we know the rip is about to be corrupted too — so we bail out before returning.

35. What a stack canary is

Concept

stack canary — A known random value the compiler places on the stack on function entry and re-checks just before return; if it changed, the program crashes BEFORE the ret runs.

Placement is the whole trick: the canary goes directly above the local variables and directly below the saved registers (the sfp and rip).

Because a contiguous overflow from a local buffer up to the rip must cross the canary first, overwriting the rip necessarily corrupts the canary — and that corruption is detected on return.

36. Picture it first: The frame, with a canary inserted

Picture it

Figure (svg): Stack frame from high to low: rip at ebp+4, sfp at ebp+0, a 4-byte canary directly below the sfp, then the 8-byte buf below that. A contiguous overflow climbing from buf to the rip must overwrite the canary on the way.

rip | sfp | canary | buf — the canary sits in the path from buf to the rip.

Discussion prompt

Read the picture before the words. What is this showing, and what is the one thing it is built to make obvious? Commit to an answer, then read on.

Hint: Name the parts, then say what changes between them — and if nothing changes, say what is being held still.

Answer:

Same vulnerable() frame from L8, now with the canary wedged between buf and the saved registers. From high to low addresses: rip, sfp, canary, then buf.

37. The frame, with a canary inserted

Concept

Same vulnerable() frame from L8, now with the canary wedged between buf and the saved registers. From high to low addresses: rip, sfp, canary, then buf.

Figure (svg): Stack frame from high to low: rip at ebp+4, sfp at ebp+0, a 4-byte canary directly below the sfp, then the 8-byte buf below that. A contiguous overflow climbing from buf to the rip must overwrite the canary on the way.

rip | sfp | canary | buf — the canary sits in the path from buf to the rip.
Slot (high→low)SizeRole
rip (saved return addr)4 Bthe attacker's prize
sfp (saved ebp)4 Bsaved register
canary4 Bguard — checked on return
buf[0..7]8 Battacker-fillable local

38. Which is which, by Size

Discrimination

Sort into buckets

Sort these by Size, from memory, without looking back at The frame, with a canary inserted. Telling them apart on the spot is the skill; the table is only where the answer happens to be written down.

4 B
rip (saved return addr); sfp (saved ebp); canary
8 B
buf[0..7]
g1
Size is "4 B" for rip (saved return addr), sfp (saved ebp), canary — that is what the table on "The frame, with a canary inserted" records, and it is the single property separating this group from the rest.
g2
Size is "8 B" for buf[0..7] — that is what the table on "The frame, with a canary inserted" records, and it is the single property separating this group from the rest.

39. The exact canary details

Concept

The book pins down five facts you must remember about the canary value itself.

40. What has to happen first: Tracing the canary catching an overflow

Ranking

Put in order

Put the moves of Tracing the canary catching an overflow into the order they have to happen.

  1. First 8 bytes fill buf
  2. Next 4 bytes overwrite the canary
  3. Bytes 13–20 reach sfp and rip
  4. On return, the canary check FAILS first

Why: These are the moves of the worked example in the order it makes them, and each one is set up by the one before it. buf[0..7] = the eight 'A's — in bounds, the canary is untouched so far.

41. Tracing the canary catching an overflow

Worked example

Run the L8 16-byte overflow into the canary-protected frame and watch the check fire.

input = "AAAAAAAA" "BBBB" "CCCC" "DDDD"   // 8 + 4 + 4 + 4 = 20 bytes
// frame (low->high): buf[8] | canary[4] | sfp[4] | rip[4]
// on return the compiler runs:  if (canary != saved_canary) abort();

First 8 bytes fill buf

Why: buf[0..7] = the eight 'A's — in bounds, the canary is untouched so far.

Next 4 bytes overwrite the canary

Why: To keep climbing toward the rip, the 'B's must land on the canary. Its value is now 0x42424242 — no longer the random secret.

Bytes 13–20 reach sfp and rip

Why: The 'C's hit the sfp and the 'D's hit the rip — exactly the hijack from L8.

On return, the canary check FAILS first

Why: Before ret runs, the compiler compares the on-stack canary (0x42424242) to the saved secret. They differ, so the program aborts — the corrupted rip is never used.

Input bytesLands onCanary intact?
AAAAAAAA (1–8)buf[0..7]yes
BBBB (9–12)canaryNO — corrupted
CCCC (13–16)sfp—
DDDD (17–20)rip— (but never reached: abort)

42. Inspect it line by line: Tracing the canary catching an overflow

Error analysis

Annotate

Walk the callouts on Tracing the canary catching an overflow. Each one is a place this is easy to get subtly wrong.

  • buf[0..7] = the eight 'A's — in bounds, the canary is untouched so far.
  • To keep climbing toward the rip, the 'B's must land on the canary. Its value is now 0x42424242 — no longer the random secret.
  • The 'C's hit the sfp and the 'D's hit the rip — exactly the hijack from L8.

43. Why the NULL byte is in the canary

Intuition

It looks like a waste to spend 8 of 32 random bits on a fixed 0x00. But that null is a second line of defense aimed squarely at string-copy overflows.

strcpy and friends stop at the first null byte. If the attacker tries to overflow through the canary with a string, their copy halts the instant it must emit the canary's null — they can't write past it to reach the rip at all. The null trades a little entropy for blocking a whole copy primitive.

44. Preview of the canary bypasses

Concept

The canary defeats a contiguous overflow of the rip. Three ideas get around it (the deep versions are L15).

45. Guess the shape of the answer: Counting the canary's entropy

Estimation

Predict first

How hard is a blind guess of the canary on a 32-bit system? Account for the NULL byte the compiler bakes in.

Commit before you compute: what does Counting the canary's entropy come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: Blind guess space is 2^24

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. About 16.7 million possibilities — large, but the null byte cost the defender a factor of 256 versus a full 2^32 secret.

46. Counting the canary's entropy

Worked example

How hard is a blind guess of the canary on a 32-bit system? Account for the NULL byte the compiler bakes in.

// 32-bit canary = 4 bytes = 32 bits total
//   1 byte (often the first) is fixed 0x00
//   remaining 3 bytes = 24 bits are random per run
guess_space = 2 ** 24      // ~16.7 million
//   vs a full 32-bit secret: 2 ** 32 (~4.3 billion)

Start from 32 bits, subtract the null byte

Why: The NULL byte is known to the attacker, so it contributes 0 bits of secrecy: 32 - 8 = 24 random bits remain.

Blind guess space is 2^24

Why: About 16.7 million possibilities — large, but the null byte cost the defender a factor of 256 versus a full 2^32 secret.

Canary detailBitsGuess space
full word322^32 ≈ 4.3 billion
minus fixed NULL byte−8—
effective entropy242^24 ≈ 16.7 million

47. Fill in: Guess space for Counting the canary's entropy

Comparison

Comparison matrix

From Counting the canary's entropy: refill the Guess space column from what you know. The rest of the table is as it appeared.

Canary detailBitsGuess space
full word322^32 ≈ 4.3 billion
minus fixed NULL byte−8—
effective entropy242^24 ≈ 16.7 million

48. Trap: 'canaries protect the heap too'

Trap

The trap

Does a stack canary protect a heap buffer overflow?

Assume the canary guards heap allocations too

Why: The name says 'stack' for a reason. There is no canary sitting next to your malloc'd buffer — a heap overflow corrupts heap metadata or adjacent objects with no canary in the path.

The fix

Does a stack canary protect a heap buffer overflow?

Stack only — the heap has no canary

Why: §4.8: the compiler inserts the canary into the stack frame, between locals and the saved registers. Heap overflows are a separate problem with separate (allocator-level) defenses.

49. Something is wrong here: 'the canary stops a format-string write'

Anomaly

Predict first

A student writes this, and it looks reasonable:

A %n format-string bug lets the attacker write to the rip directly. Does the canary catch it?

It is wrong. Say what breaks — and say it before you turn the page.

Correct: The canary only changes if it is overwritten.

A %n format-string bug lets the attacker write to the rip directly. Does the canary catch it?

Why: The canary only changes if it is overwritten. But %n writes to an arbitrary address — straight to the rip — without touching anything in between.

50. Trap: 'the canary stops a format-string write'

Trap

The trap

A %n format-string bug lets the attacker write to the rip directly. Does the canary catch it?

Assume the canary detects the rip overwrite

Why: The canary only changes if it is overwritten. But %n writes to an arbitrary address — straight to the rip — without touching anything in between.

The fix

A %n format-string bug lets the attacker write to the rip directly. Does the canary catch it?

No — the write goes AROUND the canary

Why: §4.8: the canary only detects a contiguous overflow that crosses it. A non-contiguous %n write hits the rip and leaves the canary unchanged, so the check passes and the exploit succeeds.

51. Break it on purpose: 'the canary stops a format-string write'

Break the constraint

Discussion prompt

The rule this trap just fixed:

A %n format-string bug lets the attacker write to the rip directly.

Now break it on purpose. Build a case that violates it and follow the consequences until something visibly fails. Where does the failure first show up — and would you have noticed it if you had not been looking?

Hint: The dangerous rules are the ones whose violation still produces an answer. If yours fails loudly, try to find one that fails quietly.

Answer:

The canary only changes if it is overwritten. But %n writes to an arbitrary address — straight to the rip — without touching anything in between.

52. ASLR

Section

Part 4 · §4.11

53. Move the furniture every night

Intuition

The L8 payload contained a hardcoded address — &buf, or the address of system(). That only works because the attacker knew, in advance, exactly where things would be.

ASLR is like rearranging the furniture in a dark house every single night. A burglar who memorized last night's layout now trips over everything: the addresses they wrote into the payload point at the wrong place.

54. Randomize the section base addresses

Concept

ASLR — Address Space Layout Randomization: on each run, randomize the base address of each memory section (and each loaded library).

Because each section starts at a different base every run, the absolute addresses of variables, the sfp, the rip, and the code all differ every run. The attacker can no longer hardcode an address — they must guess.

55. Why guessing an address is now a lottery

Intuition

Without ASLR, &buf is the same number every run — the attacker writes it once and it works forever. ASLR turns that one fixed answer into a different answer each run.

Now the hardcoded address in the payload is a stale guess. It was right for some past run and is almost certainly wrong for this one — so ret jumps to garbage and the program crashes instead of running the attacker's code.

56. Teach it back: Why guessing an address is now a lottery

Explain it

Discussion prompt

Explain Why guessing an address is now a lottery to a student a year behind you. No notation, no jargon they have not met — and it still has to be true.

Hint: If your explanation needs a symbol they have never seen, you are describing the notation rather than the idea.

Answer:

Without ASLR, &buf is the same number every run — the attacker writes it once and it works forever. ASLR turns that one fixed answer into a different answer each run.

57. Page boundaries limit the entropy

Concept

Randomization isn't unlimited. Sections must start on a page boundary — a multiple of the page size, typically 4096 bytes on a 32-bit system. The low bits of the base are therefore always zero.

\[ \log_2 4096 = 12 \text{ fixed low bits per address} \]

Quantity32-bit value
page size≈ 4096 bytes
fixed low bits (page offset)12
typical usable entropy≈ 16 bits → 1 in 2^16 guess

58. The key subtlety: absolute, not relative

Concept

ASLR randomizes the base of each section, which shifts every absolute address inside it by the same amount. But it does not change the relative offsets between things in the same section.

The rip is still 4 bytes above the sfp, every run. The layout within a frame is fixed; only the whole block slides.

Consequence: if the attacker can leak one absolute address (say, the sfp), they can add the known fixed offset and deduce the others (the rip, locals, etc.). One leak unlocks the section.

59. By analogy: The key subtlety: absolute, not relative

Analogy

Discussion prompt

Explain The key subtlety: absolute, not relative by analogy to something with no Computer Security in it at all — a queue, a recipe, a map, a bank balance, whatever fits. Then say where your analogy breaks.

Hint: An analogy that never breaks is not an analogy, it is the same idea wearing a hat. Find the seam — that is the part that is actually new.

Answer:

Consequence: if the attacker can leak one absolute address (say, the sfp), they can add the known fixed offset and deduce the others (the rip, locals, etc.). One leak unlocks the section.

60. Plan first: One leak defeats the randomization

Step zero

Discussion prompt

One leak defeats the randomization — before any calculation: what is the plan? Name the moves in order, in plain English, without doing the arithmetic.

Hint: It starts with: Leak one absolute address

Answer:

  1. Leak one absolute address
  2. Add the fixed relative offset
  3. Now the hardcoded-address attack works again

61. One leak defeats the randomization

Worked example

Suppose a bug leaks the runtime value of the sfp. Because relative offsets are fixed, the attacker recovers the rip's address by arithmetic — no guessing.

// relative layout is FIXED every run:
//    addr(rip) = addr(sfp) + 4
// run A: sfp leaked = 0xbf8e1240  ->  rip = 0xbf8e1244
// run B: sfp leaked = 0xbfa07c10  ->  rip = 0xbfa07c14
// bases differ, but the +4 offset never does

Leak one absolute address

Why: A separate bug (e.g. an info-leak or format-string read) discloses the runtime address of the sfp for THIS run.

Add the fixed relative offset

Why: ASLR didn't touch offsets, so addr(rip) = addr(sfp) + 4 still holds. The attacker computes the rip's address for this run.

Now the hardcoded-address attack works again

Why: With a correct per-run address in hand, the payload from L8 targets the right location. ASLR is defeated by ONE leak — no 2^16 guessing needed.

RunLeaked sfpDerived rip (sfp+4)
A0xbf8e12400xbf8e1244
B0xbfa07c100xbfa07c14
C0xbf02de800xbf02de84

62. What each one costs: One leak defeats the randomization

Trade off

Comparison matrix

From One leak defeats the randomization: every row here is a choice with a cost. Fill the Leaked sfp column, then say which row you would actually pick and what you give up for it.

RunLeaked sfpDerived rip (sfp+4)
A0xbf8e12400xbf8e1244
B0xbfa07c100xbfa07c14
C0xbf02de800xbf02de84

63. ASLR needs PIE to cover the binary

Concept

ASLR can randomize the stack, the heap, and shared libraries freely. But the program's own code can only be moved if it was compiled to be relocatable.

⊕ PIE (Position-Independent Executable) — A build flag that makes the executable's own code position-independent, so ASLR can randomize the binary's base address too — not just libraries and the stack.

Without PIE, the executable loads at a fixed address and the attacker can hardcode its functions even with ASLR on. (Flagged ⊕ — a build-time companion to ASLR.)

64. Term to definition: L12 · Exploit Mitigations Overview: NX, Stack Canaries, ASLR

Matching

Match the pairs

Match each term to the definition this lesson gave it — not the one you would guess from the word.

  • t1. mitigation
  • t2. the thesis
  • t3. stack canary
  • t4. ASLR
  • t5. ⊕ PIE (Position-Independent Executable)
  • d1. A code-hardening defense that makes a known class of exploit HARDER to pull off — typically by making a failed attempt CRASH instead of succeed.
  • d2. The only way to prevent all memory safety exploits is to use a memory-safe language.
  • d3. A known random value the compiler places on the stack on function entry and re-checks just before return; if it changed, the program crashes BEFORE the ret runs.
  • d4. Address Space Layout Randomization: on each run, randomize the base address of each memory section (and each loaded library).
  • d5. A build flag that makes the executable's own code position-independent, so ASLR can randomize the binary's base address too — not just libraries and the stack.

Why: These are the working definitions of mitigation, the thesis, stack canary, ASLR, ⊕ PIE (Position-Independent Executable) as L12 · Exploit Mitigations Overview: NX, Stack Canaries, ASLR uses them. Pairing them correctly is the test of whether you could state each one with the slide switched off.

65. Preview of the ASLR bypasses

Concept

Two ideas get past ASLR; both are L15.

66. Something is wrong here: 'ASLR randomizes relative offsets'

Anomaly

Predict first

A student writes this, and it looks reasonable:

With ASLR on, is the rip still a fixed distance from the sfp?

It is wrong. Say what breaks — and say it before you turn the page.

Correct: If offsets were randomized, leaking the sfp would tell you nothing about the rip.

With ASLR on, is the rip still a fixed distance from the sfp?

Why: If offsets were randomized, leaking the sfp would tell you nothing about the rip. But that's not how ASLR works — it slides whole sections, it doesn't reshuffle their insides.

67. Trap: 'ASLR randomizes relative offsets'

Trap

The trap

With ASLR on, is the rip still a fixed distance from the sfp?

Assume ASLR scrambles the within-frame layout too

Why: If offsets were randomized, leaking the sfp would tell you nothing about the rip. But that's not how ASLR works — it slides whole sections, it doesn't reshuffle their insides.

The fix

With ASLR on, is the rip still a fixed distance from the sfp?

Yes — rip stays 4 bytes above sfp

Why: §4.11: ASLR randomizes ABSOLUTE addresses (the section base), NOT relative offsets. The internal layout is unchanged, which is exactly why a single leaked address compromises the whole section.

68. Putting It Together

Section

Part 5 · §4.4–§4.11

69. The consolidated mitigation table

Concept

Three mitigations, three attack links, three previewed bypasses. This single table previews all of Week 5.

MitigationAttack it stopsHow it's bypassed
NX / W^Xshellcode INJECTION (run bytes in a writable buffer)code reuse — ret2libc / ROP (L13–L14)
stack canarycontiguous overflow of the ripguess (~2^24) / leak the canary / non-contiguous write (format string)
ASLRhardcoded addressesguess (~2^16) or leak an address (L15)

Notice each bypass requires the attacker to learn or reuse something they didn't have before — that's the cost the mitigation imposes.

70. Break it if you can: The consolidated mitigation table

Counterexample

Discussion prompt

Three mitigations, three attack links, three previewed bypasses. This single table previews all of Week 5.

That is stated as though it always holds. Do one of two things: produce a case where it fails, or say precisely what rules such a case out. "It just does" is not on the menu.

Hint: Hunt at the extremes first — zero, one, negative, empty, equal. If every extreme survives, the reason they survive is the proof.

Answer:

Notice each bypass requires the attacker to learn or reuse something they didn't have before — that's the cost the mitigation imposes.

71. Why all three together is the wall

Intuition

Alone, each is bypassable. Together they compose: to win against all three, the attacker must, in one exploit, leak an address (beat ASLR) and read the canary (beat the canary) and reuse existing code (beat NX).

That usually means finding two distinct bugs — a read/leak primitive and a write primitive. Doubling the bugs needed is the synergy: the whole is harder than the sum of the parts.

72. What has to happen first: What the attacker needs against all three

Ranking

Put in order

Put the moves of What the attacker needs against all three into the order they have to happen.

  1. Two of three needs are leaks
  2. The third need is a controlled write via reuse
  3. Conclusion: chain a leak with a write

Why: These are the moves of the worked example in the order it makes them, and each one is set up by the one before it. Both ASLR and the canary are beaten by disclosing a secret (an address, a value) — so the attacker needs a read/info-leak primitive.

73. What the attacker needs against all three

Worked example

Tally the prerequisites an exploit must satisfy to land on a system with NX + canary + ASLR all on. Each row is a separate thing the attacker must obtain.

// requirements to win against NX + canary + ASLR:
//   1. defeat ASLR   -> learn a real runtime address (leak)
//   2. defeat canary -> learn the secret canary value (leak)
//   3. defeat NX     -> reuse existing code via ROP (no injection)
// (1) and (2) need a READ/leak bug; (3) needs a WRITE bug
// => typically TWO distinct vulnerabilities, chained

Two of three needs are leaks

Why: Both ASLR and the canary are beaten by disclosing a secret (an address, a value) — so the attacker needs a read/info-leak primitive.

The third need is a controlled write via reuse

Why: NX forces ROP: the attacker builds the payload only from existing executable code, then needs a write primitive to place the ROP chain on the stack.

Conclusion: chain a leak with a write

Why: One bug rarely supplies both. Combining the mitigations typically forces the attacker to find and chain MULTIPLE distinct vulnerabilities — that is the synergy in concrete terms.

MitigationWhat must be learned/doneBug primitive
ASLRa real runtime addressread / leak
stack canarythe secret canary valueread / leak
NXa ROP chain from existing codewrite

74. Fill in: Bug primitive for What the attacker needs against all three

Comparison

Comparison matrix

From What the attacker needs against all three: refill the Bug primitive column from what you know. The rest of the table is as it appeared.

MitigationWhat must be learned/doneBug primitive
ASLRa real runtime addressread / leak
stack canarythe secret canary valueread / leak
NXa ROP chain from existing codewrite

75. Something is wrong here: 'one good mitigation is enough'

Anomaly

Predict first

A student writes this, and it looks reasonable:

We enabled ASLR. Can we skip the canary and NX?

It is wrong. Say what breaks — and say it before you turn the page.

Correct: Each mitigation has a known, cheap bypass: ASLR alone falls to one leak, NX alone falls to ret2libc, the canary alone falls to a format-string %n.

We enabled ASLR. Can we skip the canary and NX?

Why: Each mitigation has a known, cheap bypass: ASLR alone falls to one leak, NX alone falls to ret2libc, the canary alone falls to a format-string %n. Any single layer is a thin wall.

76. Trap: 'one good mitigation is enough'

Trap

The trap

We enabled ASLR. Can we skip the canary and NX?

Ship with only one mitigation

Why: Each mitigation has a known, cheap bypass: ASLR alone falls to one leak, NX alone falls to ret2libc, the canary alone falls to a format-string %n. Any single layer is a thin wall.

The fix

We enabled ASLR. Can we skip the canary and NX?

Enable all three — they're synergistic

Why: §4.4: the point is defense-in-depth. Combined, the attacker must defeat every layer at once (a leak AND a write), typically forcing them to chain multiple distinct vulnerabilities.

77. Which of these survive contact with L12 · Exploit Mitigations Overview: NX…?

Two truths and a lie

Sort into buckets

Some of these hold up and some are the exact mistakes this lesson is built to prevent. Sort them.

Holds up
Shellcode stays abstract as [shellcode]; we never write real attack bytes. The skill being built is knowing which defense stops which step of the L8 attack.; Mitigations assume the bug is still there and try to stop the exploitation anyway.; A mitigation raises the bar; it does not remove the vulnerability. The book is blunt about the limit.
Breaks
We turned on NX, canaries, and ASLR. Is the program now memory-safe?; With NX on, can the attacker still hijack control flow?
sound
These are stated as this lesson states them — each one survives the edge cases L12 · Exploit Mitigations Overview: NX, Stack Canaries, ASLR puts it through.
flawed
Each of these is lifted from a trap in this deck: reasonable-sounding, and wrong in a way that only shows up once you rely on it.

78. Pattern: match a mitigation to the attack step

Pattern

  1. Ask: is the attacker running INJECTED code? If so, NX / W^X faults the jump into the writable page. Bypass to expect: code reuse (ret2libc / ROP).
  2. Ask: is the rip overwrite CONTIGUOUS (climbing from a local buffer)? If so, the stack canary in the path is corrupted and detected on return. Bypass to expect: guess / leak / non-contiguous %n write.
  3. Ask: did the attacker HARDCODE an address? If so, ASLR randomizes the section base so the address is wrong each run. Bypass to expect: guess (~2^16) or leak one address (relative offsets are fixed).
  4. Remember the limits: stack canary = stack only, not heap; ASLR randomizes absolute addresses, not relative offsets; add PIE so the binary's own code is randomized.
  5. To defend a real system: enable all three — they're synergistic — but never call the program 'safe'. The only true cure is a memory-safe language.

79. Where does it stop working: Pattern: match a mitigation to the attack…

Edge cases

Discussion prompt

Pattern: match a mitigation to the attack step works on the cases you have just seen. Push it to the edge: what is the most degenerate input it still handles — empty, zero, one item, everything equal — and what is the first case where it stops being true? Name the case, not just "it breaks".

Hint: Try the smallest legal input, then the largest, then the one where two things collide. Methods are specified at their edges; the middle takes care of itself.

Answer:

  1. Ask: is the attacker running INJECTED code? If so, NX / W^X faults the jump into the writable page. Bypass to expect: code reuse (ret2libc / ROP).
  2. Ask: is the rip overwrite CONTIGUOUS (climbing from a local buffer)? If so, the stack canary in the path is corrupted and detected on return. Bypass to…
  3. Ask: did the attacker HARDCODE an address? If so, ASLR randomizes the section base so the address is wrong each run. Bypass to expect: **guess (~2^16)…
  4. Remember the limits: stack canary = stack only, not heap; ASLR randomizes absolute addresses, not relative offsets; add PIE so the binary's…
  5. To defend a real system: enable all three — they're synergistic — but never call the program 'safe'. The only true cure is a memory-safe language.

80. Rule out three: Checkpoint — why doesn't the canary catch it?

Elimination

Eliminate the wrong options

Why does the stack canary FAIL to stop this format-string write to the rip?

3 of these 4 are wrong. Strike them one at a time, and say what rules each one out before you strike the next. The survivor is the answer.

  • A. The %n write is non-contiguous — it hits the rip directly without crossing (corrupting) the canary, so the canary check still passes.
  • B. The canary protects the heap, and the rip lives on the heap, so the stack canary was never in the path.
  • C. NX already neutralized the canary, so the canary check is skipped at runtime.
  • D. ASLR randomized the canary's relative offset, so the compiler checks the wrong stack slot.

Survives elimination: A

Why: A stack canary only detects a CONTIGUOUS overflow that climbs from a local buffer up through the canary to the rip. A format-string %n write goes straight to an arbitrary address — the rip — without ever touching the canary, so the on-return comparison still matches and the program returns into attacker control.

81. Checkpoint — why doesn't the canary catch it?

Check

A 32-bit program has a stack canary, NX, and ASLR all enabled. The attacker finds a format-string bug that uses %n to write 4 bytes to an arbitrary address of their choosing, and they point it at the rip. Solve on paper before clicking.

Check your understanding

Why does the stack canary FAIL to stop this format-string write to the rip?

  • A. The %n write is non-contiguous — it hits the rip directly without crossing (corrupting) the canary, so the canary check still passes. (correct)
  • B. The canary protects the heap, and the rip lives on the heap, so the stack canary was never in the path.
  • C. NX already neutralized the canary, so the canary check is skipped at runtime.
  • D. ASLR randomized the canary's relative offset, so the compiler checks the wrong stack slot.

Answer: A

Why: A stack canary only detects a CONTIGUOUS overflow that climbs from a local buffer up through the canary to the rip. A format-string %n write goes straight to an arbitrary address — the rip — without ever touching the canary, so the on-return comparison still matches and the program returns into attacker control.

Why B tempts people
The rip is on the STACK, and the stack canary does sit in its frame — the canary's limitation here is non-contiguity, not a heap-vs-stack mismatch. (Canaries genuinely don't protect the heap, but that's irrelevant to a rip write.)
Why C tempts people
NX and the stack canary are independent mitigations; NX governs which pages are executable and never disables or skips the canary check.
Why D tempts people
ASLR randomizes ABSOLUTE addresses (section bases), not RELATIVE offsets — the canary's position within the frame is fixed, so the compiler always checks the correct slot.

82. Misconceptions to drop now

Concept

83. Synthesis — where this sits in the course

Concept

84. Primary sources & where to read more

Concept

85. Connect it up: L12 · Exploit Mitigations Overview: NX, Stack Canaries, ASLR

Connect it up

Draw it

One page, no notation unless you need it: draw how these connect — Why Mitigations Exist · Non-Executable Pages · Stack Canaries · ASLR · Putting It Together. Put an arrow wherever one of them is what makes another possible, and label the arrow with why.

86. Recap — Lesson 12

Recap

You can explain why mitigations exist (and that only a memory-safe language is a true cure), describe NX/W^X stopping shellcode injection, place a stack canary and trace it catching a contiguous overflow, describe ASLR and the absolute-vs-relative subtlety, and fill in the mitigation → attack → bypass table — arguing why combining all three is synergistic.

MitigationStopsBypassed by
NX / W^Xshellcode injectioncode reuse (ret2libc / ROP)
stack canarycontiguous rip overwriteguess / leak / non-contiguous %n write
ASLRhardcoded addressesguess (~2^16) or leak one address
the thesis—only a memory-safe language prevents ALL exploits

Sources

  1. CS 161 Computer Security Textbook §4.4 (Why mitigations), §4.5 (Non-executable pages), §4.8 (Stack canaries), §4.11 (ASLR) — Wagner, Weaver, Kao, Shakir, Law & Ngai, UC Berkeley
  2. The PaX Team, 'Address Space Layout Randomization' design documentation (2001)
  3. Cowan, Pu, Maier, et al., 'StackGuard: Automatic Adaptive Detection and Prevention of Buffer-Overflow Attacks', 7th USENIX Security Symposium (1998)
  4. OpenBSD W^X (Write XOR Execute) documentation

Want this taught 1-on-1? Alexander tutors Computer Security — $55/session, free consultation.

Book on Wyzant · Text (657) 465-8108