L14 · Return-Oriented Programming & Stack Canaries

CS 161, Lesson 14, in 50 slides and code mode. It generalizes ret2libc into ROP gadget chains, works the book's two-gadget example that adds 4 to edx, and covers finding gadgets and chaining them across rets. It then covers stack canaries from section 4.8 - a random word containing a NULL, placed between the locals and the saved registers and checked on return - and the limits of the canary that lead into Lesson 15. The examples are toys running in a sandbox, and it is anchored to textbook sections 4.7 to 4.8.

Subject: Computer Security · 81 slides · code lesson

Open the interactive version of this deck · Homework for this lesson

What this lesson covers

The lesson, slide by slide

1. Return-Oriented Programming & Canaries

Title

CS 161 · Lesson 14 of 45

ROP gadgets · chaining rets · add $0x4,%edx from pieces · stack canaries · why a canary detects but never prevents

2. By the end of this lesson you can…

Objectives

  1. Explain how ROP generalizes ret2libc: chain return addresses to run gadgets instead of whole functions.
  2. Trace the book's two-gadget chain that emulates add $0x4, %edx, and account for the clobbered EBX.
  3. Describe how a gadget finder scans a binary for byte sequences ending in ret, and how each ret is the glue.
  4. State the exact stack-canary design: a random NULL-containing word between locals and the saved sfp/rip, checked on return.
  5. List what a canary does and does not stop — the bridge to the L15 bypasses (guessing, leaks, format strings).

3. What survived from L13 · Non-Executable Pages & Return-to-libc?

Warm-up

Discussion prompt

Before we open L14 · Return-Oriented Programming & Stack Canaries: without looking back, what was the main idea of L13 · Non-Executable Pages & Return-to-libc, and what could you do by the end of it that you could not do before?

Hint: One sentence for the idea, one for the skill. If the second one is blank, that is the part to revisit.

Answer:

CS 161 Lesson 13 (54 slides, code mode): W^X / NX pages and the page-table NX bit, why the L8 buffer-injection attack now faults, the code-reuse insight that defeats NX, the 32-bit ret2libc stack layout (fake return address + argument pointer), finding the libc base with ASLR off vs. needing a leak with ASLR on, and chaining libc calls toward ROP.

4. A note before we start: this is defensive

Concept

Every example here is a toy in a sandbox, for authorized security education — exactly how the textbook teaches it. You learn the attack so you can recognize and defend against it.

We never write real shellcode; payloads stay abstract. The skill being built is reasoning about control flow and frame layout — and understanding why a defense works or doesn't.

5. Break it if you can: A note before we start: this is defensive

Counterexample

Discussion prompt

Every example here is a toy in a sandbox, for authorized security education — exactly how the textbook teaches it. You learn the attack so you can recognize and defend against it.

That is stated as though it always holds. Do one of two things: produce a case where it fails, or say precisely what rules such a case out. "It just does" is not on the menu.

Hint: Hunt at the extremes first — zero, one, negative, empty, equal. If every extreme survives, the reason they survive is the proof.

Answer:

We never write real shellcode; payloads stay abstract. The skill being built is reasoning about control flow and frame layout — and understanding why a defense works or doesn't.

6. From ret2libc to ROP

Section

Part 1 · §4.7

7. Where we left off: ret2libc

Concept

L13 defeated NX (a non-executable stack) without injecting code: overwrite the rip to return into an existing function like system. You reused code already in memory.

But returning into a whole function is coarse. What if the exact behavior you want isn't a single function lying around? ROP makes the reuse fine-grained.

8. By analogy: Where we left off: ret2libc

Analogy

Discussion prompt

Explain Where we left off: ret2libc by analogy to something with no Computer Security in it at all — a queue, a recipe, a map, a bank balance, whatever fits. Then say where your analogy breaks.

Hint: An analogy that never breaks is not an analogy, it is the same idea wearing a hat. Find the seam — that is the part that is actually new.

Answer:

L13 defeated NX (a non-executable stack) without injecting code: overwrite the rip to return into an existing function like system. You reused code already in memory.

9. Return-oriented programming

Concept

Instead of overwriting one return address to jump into one function, overwrite a chain of return addresses, starting at the rip, to execute a series of short snippets called ROP gadgets.

ROP gadget — A short sequence of instructions already in memory that ENDS IN ret. Gadgets are not functions — they need no prologue or epilogue.

You build custom shellcode out of pieces of existing code: each gadget does a tiny bit of work, then its ret hands control to the next gadget.

10. Teach it back: Return-oriented programming

Explain it

Discussion prompt

Explain Return-oriented programming to a student a year behind you. No notation, no jargon they have not met — and it still has to be true.

Hint: If your explanation needs a symbol they have never seen, you are describing the notation rather than the idea.

Answer:

You build custom shellcode out of pieces of existing code: each gadget does a tiny bit of work, then its ret hands control to the next gadget.

11. Why a gadget is not a function

Intuition

A function is a self-contained unit with a prologue (set up a frame) and an epilogue (tear it down). A gadget is just whatever instructions happen to sit right before a ret — often the tail end of some unrelated function.

You don't call a gadget; you fall into it because the previous ret pointed there. The ret at its end is the only part that matters structurally — it's the hook for the next gadget.

12. Gadgets are like a ransom note

Intuition

A ransom note is built from letters cut out of magazines: you don't write new letters, you assemble a message from fragments that already exist. ROP is the same — you splice tiny code fragments already in the binary into a program you couldn't otherwise write.

The stack you craft is the glue: each return address says 'use this fragment next.' The ret instructions are the scissors-and-tape that string them together.

13. Why ROP matters: NX is no longer a huge issue

Concept

NX assumed the attacker had to inject and run new code on the stack. ROP never injects executable code — it only reuses existing, already-executable code — so a non-executable stack doesn't stop it.

Because of ROP, the book notes that NX is no longer a huge issue for a determined attacker. The chain runs entirely from code the loader already marked executable.

14. Something is wrong here: 'a gadget is a whole function'

Anomaly

Predict first

A student writes this, and it looks reasonable:

What does each address in the ROP chain point to?

It is wrong. Say what breaks — and say it before you turn the page.

Correct: That's ret2libc, not ROP. If gadgets had to be whole functions you could only do what existing functions already do — losing all the fine-grained power.

What does each address in the ROP chain point to?

Why: That's ret2libc, not ROP. If gadgets had to be whole functions you could only do what existing functions already do — losing all the fine-grained power.

15. Trap: 'a gadget is a whole function'

Trap

The trap

What does each address in the ROP chain point to?

Assume each return address targets a complete function with its own prologue/epilogue

Why: That's ret2libc, not ROP. If gadgets had to be whole functions you could only do what existing functions already do — losing all the fine-grained power.

The fix

What does each address in the ROP chain point to?

Each points to a short instruction sequence ENDING IN ret

Why: §4.7: a gadget is a few instructions already in memory that end in ret. No prologue, no epilogue — just useful work plus the ret that advances the chain.

16. The Two-Gadget Example

Section

Part 2 · §4.7

17. Goal: emulate add $0x4, %edx

Concept

Say there is no single gadget that does add $0x4, %edx directly. The book's example builds that effect out of two gadgets found at fixed addresses in the loaded program.

The two gadgets live in functions foo and bar — but we don't call foo or bar. We jump only to the tails that interest us.

18. Why we settle for 'close enough'

Intuition

An attacker rarely finds a gadget that does exactly the operation wanted. Instead you take the gadgets that exist and compose them — even if the result lands in the wrong register or has a leftover side effect.

So the plan isn't 'find add $0x4,%edx'; it's 'find a path through available gadgets whose net effect is +4 on the value, then clean up the bookkeeping afterward.'

19. The two gadgets in loaded memory

Concept

; gadget in foo:
0x4005a1:  mov %edx, %eax
0x4005a3:  leave
0x4005a4:  ret

; gadget in bar:
0x400604:  add $0x4, %eax
0x400608:  pop %ebx
0x40060a:  leave
0x40060b:  ret

Gadget 1 (in foo) copies EDX → EAX. Gadget 2 (in bar) does EAX += 4. Run them in order and EAX ends up holding edx + 4.

GadgetEntry addressUseful effect
foo's tail0x4005a1mov %edx,%eax (EDX → EAX)
bar's tail0x400604add $0x4,%eax (EAX += 4)
both end inretpops next address, advances chain

20. Fill in: Entry address for The two gadgets in loaded memory

Comparison

Comparison matrix

From The two gadgets in loaded memory: refill the Entry address column from what you know. The rest of the table is as it appeared.

GadgetEntry addressUseful effect
foo's tail0x4005a1mov %edx,%eax (EDX → EAX)
bar's tail0x400604add $0x4,%eax (EAX += 4)
both end inretpops next address, advances chain

21. What has to happen first: Building the chain on the stack

Ranking

Put in order

Put the moves of Building the chain on the stack into the order they have to happen.

  1. On return, ret pops 0x4005a1 into eip
  2. foo's gadget ends in leave; ret pops 0x400604
  3. bar's gadget runs pop %ebx, consuming the next stack word

Why: These are the moves of the worked example in the order it makes them, and each one is set up by the one before it. The overwritten rip sends control into foo's tail.

22. Building the chain on the stack

Worked example

Overflow the buffer so the rip = 0x4005a1 (gadget 1). Above it, place 0x400604 (gadget 2), and above that a word for pop %ebx to consume.

; stack the attacker writes, low -> high (rip first):
rip      = 0x004005a1   ; -> gadget 1 (foo's tail)
next ret = 0x00400604   ; -> gadget 2 (bar's tail)
ebx word = 0x........    ; consumed by 'pop %ebx' in bar

On return, ret pops 0x4005a1 into eip

Why: The overwritten rip sends control into foo's tail. mov %edx,%eax runs: EAX now holds the old EDX.

foo's gadget ends in leave; ret pops 0x400604

Why: The next stack word is gadget 2's address. Control jumps into bar's tail. add $0x4,%eax runs: EAX is now EDX + 4.

bar's gadget runs pop %ebx, consuming the next stack word

Why: pop %ebx reads one word off the stack into EBX — so we had to place a filler word there. EBX is now clobbered; the chain must account for it.

Stack word (low→high)ValueEffect when reached
rip0x004005a1jump to foo gadget: EAX = EDX
next ret0x00400604jump to bar gadget: EAX += 4
popped into ebxfiller wordconsumed by pop %ebx; EBX clobbered

23. What each one costs: Building the chain on the stack

Trade off

Comparison matrix

From Building the chain on the stack: every row here is a choice with a cost. Fill the Value column, then say which row you would actually pick and what you give up for it.

Stack word (low→high)ValueEffect when reached
rip0x004005a1jump to foo gadget: EAX = EDX
next ret0x00400604jump to bar gadget: EAX += 4
popped into ebxfiller wordconsumed by pop %ebx; EBX clobbered

24. The bookkeeping you must track

Concept

The result ended up in EAX, not EDX — so to truly mimic add $0x4, %edx you'd need another gadget to move EAX back into EDX. The pieces rarely fit perfectly.

And pop %ebx consumed a stack word and clobbered EBX. Both side effects must be planned around: the popped word has to be placed on the stack, and EBX's old value is gone.

Tracking all of this by hand is tedious — which is exactly why ROP compilers exist.

25. ROP compilers

Concept

ROP compiler — A tool that searches a binary for gadgets and automatically chains them — handling register bookkeeping and consumed stack words — to produce a desired computation.

You describe the effect you want (e.g. 'set these registers, call this'); the compiler finds gadgets and emits the stack layout. The tedious EAX-vs-EDX and clobbered-EBX accounting is automated.

26. Take the definitions apart: ROP gadget vs ROP compiler

Definition probe

Sort into buckets

Every line below is part of the definition of ROP gadget or of ROP compiler — one or the other, never both. Put each where it belongs.

ROP gadget
A short sequence of instructions already in memory that ENDS IN ret.; they need no prologue or epilogue.
ROP compiler
A tool that searches a binary for gadgets and automatically chains them; handling register bookkeeping and consumed stack words; to produce a desired computation.
b1
A short sequence of instructions already in memory that ENDS IN ret. Gadgets are not functions — they need no prologue or epilogue.
b2
A tool that searches a binary for gadgets and automatically chains them — handling register bookkeeping and consumed stack words — to produce a desired computation.

27. Why ROP compilers had to exist

Intuition

Two gadgets already forced us to track a wrong-register result and a consumed stack word. A real payload chains dozens of gadgets — the bookkeeping explodes far past what a human wants to do by hand.

So tooling automates it, the same way a compiler turns high-level code into machine instructions. That automation is precisely why ROP went from a research curiosity to a practical, routine technique — and why NX alone stopped being enough.

28. Something is wrong here: ignoring the pop's consumed stack word

Anomaly

Predict first

A student writes this, and it looks reasonable:

Laying out the chain for the two gadgets above.

It is wrong. Say what breaks — and say it before you turn the page.

Correct: Forgets that bar's gadget runs pop %ebx, which reads a stack word.

Laying out the chain for the two gadgets above.

Why: Forgets that bar's gadget runs pop %ebx, which reads a stack word. With nothing placed there, it pops the wrong bytes and the chain derails after add.

29. Trap: ignoring the pop's consumed stack word

Trap

The trap

Laying out the chain for the two gadgets above.

Place 0x4005a1 then 0x400604 and stop

Why: Forgets that bar's gadget runs pop %ebx, which reads a stack word. With nothing placed there, it pops the wrong bytes and the chain derails after add.

The fix

Laying out the chain for the two gadgets above.

Place 0x4005a1, then 0x400604, then a filler word for the pop

Why: §4.7: every pop in a gadget consumes a stack word you must supply. Account for the popped word (and the clobbered EBX) or the chain misaligns.

30. Finding & Chaining Gadgets

Section

Part 3 · §4.7

31. Why a big codebase is dangerous

Intuition

A program linked against many libraries has an enormous amount of machine code — and therefore an enormous number of byte sequences that happen to end in ret.

With enough gadgets you can compute anything — the gadget set is effectively Turing-complete. The more code is loaded, the richer the attacker's instruction set becomes.

32. The gadget finder

Concept

gadget finder — A tool that scans a binary for byte sequences that end in a ret, cataloging each as a usable gadget along with the instructions it performs.

It looks for useful tails: on 64-bit, a classic is pop rdi; ret (load an argument register, then return); on 32-bit, pop %ebx; ret. Anything ending in ret that does something useful is a candidate.

ArchitectureExample gadgetUse
64-bitpop rdi; retload first arg register, then advance
32-bitpop %ebx; retload EBX from stack, then advance
either...; retany useful tail ending in ret

33. The ret is the glue

Concept

Recall ret pops the next stack word into eip and jumps there. In a normal program that returns to the caller; in a ROP chain it advances to the next gadget.

So the chain runs itself: each gadget does its work, hits ret, pops the next address the attacker planted, and continues. The stack is the program; the rets are the instruction pointer stepping through it.

34. Where does each piece belong: L14 · Return-Oriented Programming & Stack…

Sorting

Sort into buckets

These are the pieces of L14 · Return-Oriented Programming & Stack Canaries, out of order. Put each one back under the part of the lesson it belongs to.

From ret2libc to ROP
Where we left off: ret2libc; Return-oriented programming; Why a gadget is not a function
The Two-Gadget Example
Goal: emulate add $0x4, %edx; Why we settle for 'close enough'; The two gadgets in loaded memory
Finding & Chaining Gadgets
Why a big codebase is dangerous; The gadget finder; The ret is the glue
s1
From ret2libc to ROP is where L14 · Return-Oriented Programming & Stack Canaries puts Where we left off: ret2libc, Return-oriented programming, Why a gadget is not a function. Knowing which part of the lesson a problem belongs to is most of knowing which method to reach for.
s2
The Two-Gadget Example is where L14 · Return-Oriented Programming & Stack Canaries puts Goal: emulate add $0x4, %edx, Why we settle for 'close enough', The two gadgets in loaded memory. Knowing which part of the lesson a problem belongs to is most of knowing which method to reach for.
s3
Finding & Chaining Gadgets is where L14 · Return-Oriented Programming & Stack Canaries puts Why a big codebase is dangerous, The gadget finder, The ret is the glue. Knowing which part of the lesson a problem belongs to is most of knowing which method to reach for.

35. The stack is the program

Intuition

Normally the code is the program and the stack just holds data. In ROP that flips: the attacker's stack of addresses is the program, and the existing code is a fixed instruction set you index into.

Each ret is like a tiny fetch-decode-execute step: fetch the next address from the stack, jump there, run that gadget, repeat. The 'program counter' is really esp marching up the planted chain.

36. Plan first: Tracing eip and esp across two rets

Step zero

Discussion prompt

Tracing eip and esp across two rets — before any calculation: what is the plan? Name the moves in order, in plain English, without doing the arithmetic.

Hint: It starts with: Function returns: ret pops 0x1000, esp += 4, eip = 0x1000

Answer:

  1. Function returns: ret pops 0x1000, esp += 4, eip = 0x1000
  2. Gadget A finishes with ret: pops 0x2000, esp += 4, eip = 0x2000
  3. Gadget B's ret pops the next word and continues

37. Tracing eip and esp across two rets

Worked example

Two trivial gadgets: gadget A at 0x1000 ends in ret; gadget B at 0x2000 ends in ret. The attacker's stack holds A's address at the rip, then B's address just above it.

; attacker stack (low -> high):
esp -> 0x00001000   ; A's address (the overwritten rip)
        0x00002000   ; B's address
        ........      ; whatever B's ret pops next

Function returns: ret pops 0x1000, esp += 4, eip = 0x1000

Why: The overwritten rip is popped into eip; esp now points at B's address. Control runs gadget A.

Gadget A finishes with ret: pops 0x2000, esp += 4, eip = 0x2000

Why: A's ret reads the word esp points at (B's address), advances esp, and jumps. Control runs gadget B.

Gadget B's ret pops the next word and continues

Why: The same mechanism repeats: each ret consumes one address and steps to the next gadget. esp marches up the chain.

Stepesp points ateip after ret
start (return)A's address (0x1000)0x1000 — gadget A
after A's retB's address (0x2000)0x2000 — gadget B
after B's retnext chain wordnext gadget

38. Fill in: eip after ret for Tracing eip and esp across two rets

Comparison

Comparison matrix

From Tracing eip and esp across two rets: refill the eip after ret column from what you know. The rest of the table is as it appeared.

Stepesp points ateip after ret
start (return)A's address (0x1000)0x1000 — gadget A
after A's retB's address (0x2000)0x2000 — gadget B
after B's retnext chain wordnext gadget

39. Something is wrong here: 'ROP needs you to inject code'

Anomaly

Predict first

A student writes this, and it looks reasonable:

How does the CPU end up running the ROP gadgets?

It is wrong. Say what breaks — and say it before you turn the page.

Correct: Then a non-executable stack (NX) would block it — and ROP's whole point is to defeat NX.

How does the CPU end up running the ROP gadgets?

Why: Then a non-executable stack (NX) would block it — and ROP's whole point is to defeat NX. Injected stack code is the old shellcode model, not ROP.

40. Trap: 'ROP needs you to inject code'

Trap

The trap

How does the CPU end up running the ROP gadgets?

Assume the gadgets' machine code is injected onto the stack

Why: Then a non-executable stack (NX) would block it — and ROP's whole point is to defeat NX. Injected stack code is the old shellcode model, not ROP.

The fix

How does the CPU end up running the ROP gadgets?

Only ADDRESSES are written to the stack; the code already lives in executable memory

Why: §4.7: the attacker writes a chain of return addresses. The gadget instructions are existing, already-executable code — so NX never applies to them.

41. Stack Canaries

Section

Part 4 · §4.8

42. The canary in the coal mine

Intuition

Miners once carried a caged canary underground. The bird is more sensitive to toxic gas than people are: if the canary dies, you evacuate — it is a sacrificial early warning.

A stack canary plays the same role: a sacrificial value placed where an overflow would pass through. If it's been disturbed on the way to the rip, the program knows it's been attacked and bails out.

43. What the compiler inserts

Concept

stack canary — A known random value the compiler places between a function's local variables and its saved registers (sfp and rip), checked before the function returns.

It sits directly above the locals and directly below the sfp and rip. The function never uses it for computation — it exists only to be checked on the way out.

Modern compilers add it automatically, and the runtime overhead is negligible.

44. Term to definition: L14 · Return-Oriented Programming & Stack Canaries

Matching

Match the pairs

Match each term to the definition this lesson gave it — not the one you would guess from the word.

  • t1. ROP gadget
  • t2. ROP compiler
  • t3. gadget finder
  • t4. stack canary
  • d1. A short sequence of instructions already in memory that ENDS IN ret. Gadgets are not functions — they need no prologue or epilogue.
  • d2. A tool that searches a binary for gadgets and automatically chains them — handling register bookkeeping and consumed stack words — to produce a desired computation.
  • d3. A tool that scans a binary for byte sequences that end in a ret, cataloging each as a usable gadget along with the instructions it performs.
  • d4. A known random value the compiler places between a function's local variables and its saved registers (sfp and rip), checked before the function returns.

Why: These are the working definitions of ROP gadget, ROP compiler, gadget finder, stack canary as L14 · Return-Oriented Programming & Stack Canaries uses them. Pairing them correctly is the test of whether you could state each one with the slide switched off.

45. Picture it first: Where the canary lives in the frame

Picture it

Figure (svg): Stack frame high to low: rip at ebp+4, sfp at ebp+0, then the canary directly above the locals, then buf. A contiguous overflow from buf upward must cross the canary before reaching the sfp and rip.

rip | sfp | canary | buf — a contiguous overflow must cross the canary to reach the rip.

Discussion prompt

Read the picture before the words. What is this showing, and what is the one thing it is built to make obvious? Commit to an answer, then read on.

Hint: Name the parts, then say what changes between them — and if nothing changes, say what is being held still.

Answer:

From high to low addresses: the rip, the sfp, then the canary, then the locals (buf). A contiguous overflow climbing from buf toward the rip must pass through the canary.

46. Where the canary lives in the frame

Concept

From high to low addresses: the rip, the sfp, then the canary, then the locals (buf). A contiguous overflow climbing from buf toward the rip must pass through the canary.

Figure (svg): Stack frame high to low: rip at ebp+4, sfp at ebp+0, then the canary directly above the locals, then buf. A contiguous overflow from buf upward must cross the canary before reaching the sfp and rip.

rip | sfp | canary | buf — a contiguous overflow must cross the canary to reach the rip.
Slot (high→low)SizeRole
rip (saved return addr)4 Bthe target an attacker wants
sfp (saved ebp)4 Bsaved frame pointer
canary4 Bsacrificial guard, checked on return
buf[0..7]8 Blocals — attacker-fillable

47. Which is which, by Size

Discrimination

Sort into buckets

Sort these by Size, from memory, without looking back at Where the canary lives in the frame. Telling them apart on the spot is the skill; the table is only where the answer happens to be written down.

4 B
rip (saved return addr); sfp (saved ebp); canary
8 B
buf[0..7]
g1
Size is "4 B" for rip (saved return addr), sfp (saved ebp), canary — that is what the table on "Where the canary lives in the frame" records, and it is the single property separating this group from the rest.
g2
Size is "8 B" for buf[0..7] — that is what the table on "Where the canary lives in the frame" records, and it is the single property separating this group from the rest.

48. The exact canary properties

Concept

49. Why random, and why per-run

Intuition

If the canary were a fixed constant, the attacker would just include that constant in the overflow and overwrite the canary with its correct value — the check would pass and the defense would be useless.

Generating it randomly at runtime, fresh each run, means the attacker can't bake it into the payload. To get past it they must learn this run's value — which is exactly what the L15 leak/guess attacks try to do.

50. Why a NULL byte in the canary

Intuition

Many overflows come from string copies like strcpy, which stop at the first NULL byte. If the canary's first byte is 0x00, an attacker trying to overwrite the canary with a copied string can't get past that null — the copy halts.

The cost is a little entropy: one byte is fixed at zero, so a 32-bit canary really has only about 24 bits an attacker must guess. The trade is worth it — it blocks the most common overflow primitive outright.

51. What has to happen first: An overflow trips the canary

Ranking

Put in order

Put the moves of An overflow trips the canary into the order they have to happen.

  1. First 8 bytes fill buf
  2. Next 4 bytes overwrite the canary
  3. On return, the function compares the canary; it no longer matches
  4. The program crashes BEFORE control reaches the rip

Why: These are the moves of the worked example in the order it makes them, and each one is set up by the one before it. buf[0..7] = the eight 'A's — still in bounds, canary untouched so far.

52. An overflow trips the canary

Worked example

The attacker tries the L8 trick: 8 bytes to fill buf, then more to reach the rip. But now the canary sits in between.

input = "AAAAAAAA"  "BBBB"  "CCCC"  "DDDD"
;        buf[0..7]  canary  sfp    rip
; the copy must overwrite the canary (BBBB) to reach sfp/rip

First 8 bytes fill buf

Why: buf[0..7] = the eight 'A's — still in bounds, canary untouched so far.

Next 4 bytes overwrite the canary

Why: To reach the sfp and rip the contiguous write MUST cross the canary slot — replacing the random value with 'BBBB'.

On return, the function compares the canary; it no longer matches

Why: The stored 'BBBB' differs from the random value the function saved. The mismatch is detected.

The program crashes BEFORE control reaches the rip

Why: §4.8: the check runs before ret. Because the canary changed, the program aborts — the overwritten rip is never used, so the hijack fails.

Input bytesLands onResult
AAAAAAAA (0–7)buf[0..7]in bounds
BBBB (8–11)canarycorrupted — mismatch on return
CCCC / DDDD (12–19)sfp / ripoverwritten, but never reached: crash first

53. What each one costs: An overflow trips the canary

Trade off

Comparison matrix

From An overflow trips the canary: every row here is a choice with a cost. Fill the Lands on column, then say which row you would actually pick and what you give up for it.

Input bytesLands onResult
AAAAAAAA (0–7)buf[0..7]in bounds
BBBB (8–11)canarycorrupted — mismatch on return
CCCC / DDDD (12–19)sfp / ripoverwritten, but never reached: crash first

54. Something is wrong here: 'the canary stops the overflow from happening'

Anomaly

Predict first

A student writes this, and it looks reasonable:

Believe the canary blocks the out-of-bounds write itself

It is wrong. Say what breaks — and say it before you turn the page.

Correct: It doesn't. The overflow already happened — buf, the canary, the sfp, and the rip are all overwritten in memory.

When does the canary act?

Why: It doesn't. The overflow already happened — buf, the canary, the sfp, and the rip are all overwritten in memory. Nothing stopped the write.

55. Trap: 'the canary stops the overflow from happening'

Trap

The trap

When does the canary act?

Believe the canary blocks the out-of-bounds write itself

Why: It doesn't. The overflow already happened — buf, the canary, the sfp, and the rip are all overwritten in memory. Nothing stopped the write.

The fix

When does the canary act?

The canary is DETECTED on return, after the write

Why: §4.8: the corruption occurs, then on the way out the function checks the canary and crashes. It prevents the hijack (the rip is never used), not the overflow.

56. What Canaries Miss

Section

Part 5 · §4.8 → L15

57. Canaries only guard one path

Concept

A canary protects exactly the contiguous-overflow-to-rip path: a write that climbs from the locals, through the canary, into the saved registers. That is real protection — but it is the only path it covers.

Anything that reaches sensitive memory without crossing the canary sails right past it.

58. A tripwire across one hallway

Intuition

Picture the canary as a tripwire stretched across one corridor — the contiguous path from the locals up to the saved rip. Anyone walking that corridor trips it.

But the building has other ways in: a window onto the rip (a %n write), a side room of locals below the wire (an authenticated flag), and an entirely separate building (the heap). The tripwire guards none of those.

59. Teach it back: A tripwire across one hallway

Explain it

Discussion prompt

Explain A tripwire across one hallway to a student a year behind you. No notation, no jargon they have not met — and it still has to be true.

Hint: If your explanation needs a symbol they have never seen, you are describing the notation rather than the idea.

Answer:

Picture the canary as a tripwire stretched across one corridor — the contiguous path from the locals up to the saved rip. Anyone walking that corridor trips it.

60. What has to happen first: Overwriting an adjacent local, not the rip

Ranking

Put in order

Put the moves of Overwriting an adjacent local, not the rip into the order they have to happen.

  1. Overflow buf just enough to overwrite authenticated
  2. Set authenticated to a nonzero value
  3. On return, the canary still matches — no crash

Why: These are the moves of the worked example in the order it makes them, and each one is set up by the one before it. The write stops below the canary — it never reaches or disturbs the canary slot.

61. Overwriting an adjacent local, not the rip

Worked example

Suppose a frame has char buf[8] and, just above it but below the canary, an int authenticated flag. The attacker doesn't need the rip at all.

; frame (high -> low): rip | sfp | canary | authenticated | buf
input = "AAAAAAAA"  "\x01\x00\x00\x00"
;        buf[0..7]   authenticated = 1   (canary never touched)

Overflow buf just enough to overwrite authenticated

Why: The write stops below the canary — it never reaches or disturbs the canary slot.

Set authenticated to a nonzero value

Why: The logic now treats the attacker as logged in, even though they never crossed the canary.

On return, the canary still matches — no crash

Why: The canary was untouched, so the check passes. The attack succeeded without ever going near the rip.

Target overwrittenCrosses canary?Canary catches it?
the rip (contiguous)yesyes — crash
adjacent local below canarynono — undetected
heap datan/a (not on stack)no — undetected

62. Fill in: Crosses canary? for Overwriting an adjacent local, not the rip

Comparison

Comparison matrix

From Overwriting an adjacent local, not the rip: refill the Crosses canary? column from what you know. The rest of the table is as it appeared.

Target overwrittenCrosses canary?Canary catches it?
the rip (contiguous)yesyes — crash
adjacent local below canarynono — undetected
heap datan/a (not on stack)no — undetected

63. Three things a canary does NOT stop

Concept

64. Why %n walks around the canary

Intuition

A buffer overflow is contiguous: it writes byte after byte, so it can't skip the canary on its way up. A format-string %n is arbitrary: it writes a chosen value to a chosen address.

So %n can drop a value directly onto the rip's 4 bytes without touching the canary slot at all. The canary is a tripwire across one hallway; %n comes through the window.

65. By analogy: Why %n walks around the canary

Analogy

Discussion prompt

Explain Why %n walks around the canary by analogy to something with no Computer Security in it at all — a queue, a recipe, a map, a bank balance, whatever fits. Then say where your analogy breaks.

Hint: An analogy that never breaks is not an analogy, it is the same idea wearing a hat. Find the seam — that is the part that is actually new.

Answer:

A buffer overflow is contiguous: it writes byte after byte, so it can't skip the canary on its way up. A format-string %n is arbitrary: it writes a chosen value to a chosen address.

66. Something is wrong here: 'a canary stops a format-string %n attack'

Anomaly

Predict first

A student writes this, and it looks reasonable:

Does a stack canary defend against a %n write to the rip?

It is wrong. Say what breaks — and say it before you turn the page.

Correct: It only guards the contiguous path.

Does a stack canary defend against a %n write to the rip?

Why: It only guards the contiguous path. %n writes to an arbitrary address directly, never crossing the canary, so the canary check still passes.

67. Trap: 'a canary stops a format-string %n attack'

Trap

The trap

Does a stack canary defend against a %n write to the rip?

Assume the canary protects the rip from all writes

Why: It only guards the contiguous path. %n writes to an arbitrary address directly, never crossing the canary, so the canary check still passes.

The fix

Does a stack canary defend against a %n write to the rip?

No — %n writes around the canary to the rip directly

Why: §4.8: canaries only catch contiguous overflows. A non-contiguous write (format string) hits the rip without disturbing the canary — undetected.

68. Which of these survive contact with L14 · Return-Oriented Programming & Stack…?

Two truths and a lie

Sort into buckets

Some of these hold up and some are the exact mistakes this lesson is built to prevent. Sort them.

Holds up
You build custom shellcode out of pieces of existing code: each gadget does a tiny bit of work, then its ret hands control to the next gadget.; The stack you craft is the glue: each return address says 'use this fragment next.' The ret instructions are the scissors-and-tape that string them together.; Because of ROP, the book notes that NX is no longer a huge issue for a determined attacker. The chain runs entirely from code the loader already marked executable.
Breaks
What does each address in the ROP chain point to?; Laying out the chain for the two gadgets above.
sound
These are stated as this lesson states them — each one survives the edge cases L14 · Return-Oriented Programming & Stack Canaries puts it through.
flawed
Each of these is lifted from a trap in this deck: reasonable-sounding, and wrong in a way that only shows up once you rely on it.

69. Preview: the L15 bypasses

Concept

Even on the contiguous path, a canary can be defeated. The book previews two routes that L15 develops:

Both turn the canary from a wall into a speed bump — raising the bar, not closing the door.

70. Break it if you can: Preview: the L15 bypasses

Counterexample

Discussion prompt

Even on the contiguous path, a canary can be defeated. The book previews two routes that L15 develops:

That is stated as though it always holds. Do one of two things: produce a case where it fails, or say precisely what rules such a case out. "It just does" is not on the menu.

Hint: Hunt at the extremes first — zero, one, negative, empty, equal. If every extreme survives, the reason they survive is the proof.

Answer:

Both turn the canary from a wall into a speed bump — raising the bar, not closing the door.

71. Consolidate

Section

Part 6

72. Without one step: Pattern: reason about ROP and canaries

Constraint

Discussion prompt

Run Pattern: reason about ROP and canaries with this step confiscated:

ROP defeats NX because it reuses existing executable code; nothing is injected onto the stack.

Is it still possible? If it is, say what takes its place and what it costs you. If it is not, say exactly what that step was providing that nothing else does.

Hint: A step you can drop for free was never load-bearing. If you cannot drop it, name the thing that goes wrong the moment it is gone.

Answer:

  1. ROP generalizes ret2libc: chain return addresses to gadgets (instruction tails ending in ret), not whole functions.
  2. Each ret is the glue: it pops the next address the attacker planted and jumps there, stepping through the chain.
  3. Account for side effects: every pop consumes a stack word and clobbers a register — or use a ROP compiler.
  4. ROP defeats NX because it reuses existing executable code; nothing is injected onto the stack.
  5. A canary is a random NULL-containing word between locals and the saved sfp/rip, checked on return — it detects a contiguous overflow, it does not…
  6. Know the gaps: canaries miss heap corruption, adjacent-local overwrites, and non-contiguous %n writes; they can be guessed (~2^24) or leaked.

73. Pattern: reason about ROP and canaries

Pattern

  1. ROP generalizes ret2libc: chain return addresses to gadgets (instruction tails ending in ret), not whole functions.
  2. Each ret is the glue: it pops the next address the attacker planted and jumps there, stepping through the chain.
  3. Account for side effects: every pop consumes a stack word and clobbers a register — or use a ROP compiler.
  4. ROP defeats NX because it reuses existing executable code; nothing is injected onto the stack.
  5. A canary is a random NULL-containing word between locals and the saved sfp/rip, checked on return — it detects a contiguous overflow, it does not prevent the write.
  6. Know the gaps: canaries miss heap corruption, adjacent-local overwrites, and non-contiguous %n writes; they can be guessed (~2^24) or leaked.

74. Where does it stop working: Pattern: reason about ROP and canaries

Edge cases

Discussion prompt

Pattern: reason about ROP and canaries works on the cases you have just seen. Push it to the edge: what is the most degenerate input it still handles — empty, zero, one item, everything equal — and what is the first case where it stops being true? Name the case, not just "it breaks".

Hint: Try the smallest legal input, then the largest, then the one where two things collide. Methods are specified at their edges; the middle takes care of itself.

Answer:

  1. ROP generalizes ret2libc: chain return addresses to gadgets (instruction tails ending in ret), not whole functions.
  2. Each ret is the glue: it pops the next address the attacker planted and jumps there, stepping through the chain.
  3. Account for side effects: every pop consumes a stack word and clobbers a register — or use a ROP compiler.
  4. ROP defeats NX because it reuses existing executable code; nothing is injected onto the stack.
  5. A canary is a random NULL-containing word between locals and the saved sfp/rip, checked on return — it detects a contiguous overflow, it does not…
  6. Know the gaps: canaries miss heap corruption, adjacent-local overwrites, and non-contiguous %n writes; they can be guessed (~2^24) or leaked.

75. Rule out three: Checkpoint — what advances the chain?

Elimination

Eliminate the wrong options

In a ROP chain, what advances execution from one gadget to the next?

3 of these 4 are wrong. Strike them one at a time, and say what rules each one out before you strike the next. The survivor is the answer.

  • A. A call instruction at the start of each gadget that invokes the next one.
  • B. The ret at the end of each gadget, which pops the next address off the stack into eip and jumps there.
  • C. A jmp hard-coded into every gadget that points directly at the following gadget's address.
  • D. The CPU automatically executes whatever instruction sits at the next higher address in memory.

Survives elimination: B

Why: Each gadget ends in ret. A ret pops the next stack word into eip and jumps there — so the attacker-planted addresses on the stack drive the chain, one gadget per ret. The ret is the glue that strings gadgets together.

76. Checkpoint — what advances the chain?

Check

A ROP chain has run gadget A and now needs to reach gadget B, whose address the attacker placed on the stack just above A's. Think about the mechanism before clicking.

Check your understanding

In a ROP chain, what advances execution from one gadget to the next?

  • A. A call instruction at the start of each gadget that invokes the next one.
  • B. The ret at the end of each gadget, which pops the next address off the stack into eip and jumps there. (correct)
  • C. A jmp hard-coded into every gadget that points directly at the following gadget's address.
  • D. The CPU automatically executes whatever instruction sits at the next higher address in memory.

Answer: B

Why: Each gadget ends in ret. A ret pops the next stack word into eip and jumps there — so the attacker-planted addresses on the stack drive the chain, one gadget per ret. The ret is the glue that strings gadgets together.

Why A tempts people
Gadgets are not called — they need no prologue/epilogue and start with no call. ROP relies on ret, which is exactly why it's called return-oriented.
Why C tempts people
Gadgets aren't modified to point at each other; they're existing code. The chain order lives entirely in the attacker's stack of addresses, consumed by each ret.
Why D tempts people
Execution isn't linear by memory address here. After a gadget's ret, control goes to whatever address the attacker placed on the stack — which can be anywhere in loaded code.

77. Misconceptions to drop now

Concept

78. Synthesis — where this sits in the course

Concept

79. Primary sources & where to read more

Concept

80. Connect it up: L14 · Return-Oriented Programming & Stack Canaries

Connect it up

Draw it

One page, no notation unless you need it: draw how these connect — From ret2libc to ROP · The Two-Gadget Example · Finding & Chaining Gadgets · Stack Canaries · What Canaries Miss · Consolidate. Put an arrow wherever one of them is what makes another possible, and label the arrow with why.

81. Recap — Lesson 14

Recap

You can explain how ROP chains gadgets to generalize ret2libc and defeat NX, trace the two-gadget add-4 example (and its EAX/EBX bookkeeping), describe the gadget finder and the ret-as-glue mechanism, state the exact stack-canary design, and list precisely what a canary detects and what it misses.

IdeaThe one fact
ROP gadgetinstruction tail ending in ret; not a function
What advances the chaineach gadget's ret pops the next address
Why ROP beats NXreuses existing code; injects no stack code
Stack canaryrandom NULL-containing word between locals and sfp/rip
Canary timingDETECTS a contiguous overflow on return; never prevents the write
Canary gapsheap, adjacent locals, %n writes; guess (~2^24) or leak

Sources

  1. CS 161 Computer Security Textbook §4.7 (Return-oriented programming), §4.8 (Stack canaries) — Wagner, Weaver, Kao, Shakir, Law & Ngai, UC Berkeley
  2. Hovav Shacham, 'The Geometry of Innocent Flesh on the Bone: Return-into-libc without Function Calls (on the x86)', ACM CCS (2007) — the foundational ROP paper — ACM Conference on Computer and Communications Security, 2007
  3. Cowan, Pu, Maier, Walpole, Bakke, Beattie, Grier, Wagle & Zhang, 'StackGuard: Automatic Adaptive Detection and Prevention of Buffer-Overflow Attacks', USENIX Security (1998) — stack canaries — 7th USENIX Security Symposium, 1998
  4. CWE-121: Stack-based Buffer Overflow

Want this taught 1-on-1? Alexander tutors Computer Security — $55/session, free consultation.

Book on Wyzant · Text (657) 465-8108