CS 161, Lesson 6 and Quiz 6, in 50 slides and code mode. It covers the full 11-step x86 cdecl call and return, the prologue and epilogue, ebp-relative addressing with arguments at ebp+8, and the assembly for foo(1,2) with leave and ret. It includes two full call-trace tables and is anchored to textbook sections 2.7 to 2.9.
Subject: Computer Security · 42 slides · code lesson
Open the interactive version of this deck · Homework for this lesson
Title
CS 161 · Lesson 6 of 45 · Quiz day
the 11 steps · prologue & epilogue · args at ebp+8 · leave / ret
Objectives
push/mov/sub/leave/ret to the exact step each performs.Warm-up
Discussion prompt
Before we open L06 · The 11-Step Calling Convention & Stack-Frame Anatomy: without looking back, what was the main idea of L05 · Little-Endian, the Registers eip/ebp/esp, and Push/Pop, and what could you do by the end of it that you could not do before?
Hint: One sentence for the idea, one for the skill. If the second one is blank, that is the part to revisit.
Answer:
CS 161, Lesson 5, in 50 slides and code mode. It covers little-endian byte order, the three special 32-bit registers eip, ebp, and esp, and push and pop as decrement-then-write and read-then-increment, with a full esp trace table. It is anchored to textbook sections 2.4 to 2.6.
Section
Part 1 · §2.7
Concept
This course uses AT&T syntax (what gdb shows): the destination comes last. Registers take %, immediates take $, and parentheses dereference memory.
movl %esp, %ebp # ebp = esp (dest is last)
subl $16, %esp # esp = esp - 16 ($ = immediate)
movl 8(%ebp), %eax # eax = *(ebp + 8) (memory deref + offset)| Operand | Means |
|---|---|
| %esp | the register esp |
| $16 | the constant 16 |
| (%esp) | memory at address esp |
| 8(%ebp) | memory at address ebp + 8 |
Comparison
Comparison matrix
From AT&T syntax and addressing modes: refill the Means column from what you know. The rest of the table is as it appeared.
| Operand | Means |
|---|---|
| %esp | the register esp |
| $16 | the constant 16 |
| (%esp) | memory at address esp |
| 8(%ebp) | memory at address ebp + 8 |
Concept
This is the convention to burn in: in 32-bit x86, arguments are pushed onto the stack, in reverse order — not passed in registers. (That's the 64-bit world; we visit it once, in Lesson 7.)
Return value comes back in eax. The caller cleans up the pushed arguments after the call returns.
Because args are on the stack right next to the saved return address, attacker-controlled data and control-flow metadata sit inches apart — the seed of stack smashing.
Counterexample
Discussion prompt
Return value comes back in eax. The caller cleans up the pushed arguments after the call returns.
That is stated as though it always holds. Do one of two things: produce a case where it fails, or say precisely what rules such a case out. "It just does" is not on the menu.
Hint: Hunt at the extremes first — zero, one, negative, empty, equal. If every extreme survives, the reason they survive is the proof.
Answer:
Because args are on the stack right next to the saved return address, attacker-controlled data and control-flow metadata sit inches apart — the seed of stack smashing.
Section
Part 2 · §2.8
Concept
Calling foo from main means: point eip at foo's code, and build a new frame so ebp/esp describe foo, not main. Anything we overwrite, we first save on the stack so we can restore it on return.
| Register | Before call | Must become |
|---|---|---|
| eip | in main's code | start of foo |
| ebp | top of main's frame | top of foo's frame |
| esp | bottom of main's frame | bottom of foo's frame |
Saved copies get names: the saved eip is the rip (return instruction pointer); the saved ebp is the sfp (saved frame pointer).
Trade off
Comparison matrix
From A call must update all three registers: every row here is a choice with a cost. Fill the Before call column, then say which row you would actually pick and what you give up for it.
| Register | Before call | Must become |
|---|---|---|
| eip | in main's code | start of foo |
| ebp | top of main's frame | top of foo's frame |
| esp | bottom of main's frame | bottom of foo's frame |
Pattern
Predict first
The table runs: 1 (push 2, push 1) | −8 | arg2, arg1 on stack · 2–3 (call) | −4 | rip pushed, eip = &foo · 4 (push ebp) | −4 | sfp pushed · 5 (mov) | 0 | ebp = esp (frame anchored)
In Steps 1–6: call + prologue (main calls foo(1,2)), given the rows so far: what is the next one — the row where After step is 6 (sub N)?
Correct: 6 (sub N) | −N | locals allocated
| After step | esp moves | Pushed / set |
|---|---|---|
| 1 (push 2, push 1) | −8 | arg2, arg1 on stack |
| 2–3 (call) | −4 | rip pushed, eip = &foo |
| 4 (push ebp) | −4 | sfp pushed |
| 5 (mov) | 0 | ebp = esp (frame anchored) |
| 6 (sub N) | −N | locals allocated |
Why: The relationship between the columns, not the individual numbers, is what generates the next row. main pushes 2 then 1 (so arg1 ends up at the lower address), then call foo pushes the return address (rip) and jumps.
Worked example
1. push args (reverse): push 2, then push 1
2. push old eip -> rip } these two
3. set eip = &foo } are 'call foo'
4. push old ebp -> sfp }
5. ebp = esp } prologue
6. esp -= N (locals) }Steps 1–3 happen in the CALLER
Why: main pushes 2 then 1 (so arg1 ends up at the lower address), then call foo pushes the return address (rip) and jumps.
Steps 4–6 are the PROLOGUE, in the callee
Why: foo saves the old ebp (sfp), sets ebp = esp to anchor its frame, then drops esp by N to allocate locals.
| After step | esp moves | Pushed / set |
|---|---|---|
| 1 (push 2, push 1) | −8 | arg2, arg1 on stack |
| 2–3 (call) | −4 | rip pushed, eip = &foo |
| 4 (push ebp) | −4 | sfp pushed |
| 5 (mov) | 0 | ebp = esp (frame anchored) |
| 6 (sub N) | −N | locals allocated |
Comparison
Comparison matrix
From Steps 1–6: call + prologue (main calls foo(1,2)): refill the esp moves column from what you know. The rest of the table is as it appeared.
| After step | esp moves | Pushed / set |
|---|---|---|
| 1 (push 2, push 1) | −8 | arg2, arg1 on stack |
| 2–3 (call) | −4 | rip pushed, eip = &foo |
| 4 (push ebp) | −4 | sfp pushed |
| 5 (mov) | 0 | ebp = esp (frame anchored) |
| 6 (sub N) | −N | locals allocated |
Ranking
Put in order
Put the moves of Steps 7–11: execute + epilogue + return into the order they have to happen.
Why: These are the moves of the worked example in the order it makes them, and each one is set up by the one before it. Setting esp = ebp jumps the stack pointer back up to the frame's top, erasing the locals in one move.
Worked example
7. execute body (args at ebp+8, locals below ebp)
8. esp = ebp } epilogue
9. pop sfp -> ebp } (8+9 = leave)
10. pop rip -> eip } (10 = ret)
11. caller: esp += 8 (remove the arguments)Step 8 discards locals
Why: Setting esp = ebp jumps the stack pointer back up to the frame's top, erasing the locals in one move.
Steps 9–10 restore the caller
Why: Pop the sfp back into ebp (restoring main's frame), then pop the rip back into eip (resuming main right after the call).
| After step | esp moves | Restored |
|---|---|---|
| 8 (esp = ebp) | +N | locals gone |
| 9 (pop ebp) | +4 | ebp = main's frame (sfp) |
| 10 (ret) | +4 | eip = main's code (rip) |
| 11 (add $8) | +8 | arguments removed (caller) |
Verify esp is back where it started
Why: Total down in steps 1–6 (8+4+4+N) exactly equals total up in steps 8–11 (N+4+4+8). esp returns to its pre-call value — which is why esp never needs saving.
Discrimination
Sort into buckets
Sort these by esp moves, from memory, without looking back at Steps 7–11: execute + epilogue + return. Telling them apart on the spot is the skill; the table is only where the answer happens to be written down.
Concept
Once ebp is anchored (step 5), every interesting slot is a fixed offset from ebp. Memorize this picture — exam questions and the overflow math both depend on it.
high addr
arg2 ebp + 12
arg1 ebp + 8 <- first argument
rip (ret) ebp + 4
sfp (old ebp)ebp + 0 <- ebp points here
local ... ebp - 4
local ... esp <- bottom
low addr| Slot | Offset from ebp |
|---|---|
| first argument | ebp + 8 |
| saved return addr (rip) | ebp + 4 |
| saved frame ptr (sfp) | ebp + 0 |
| first local | ebp − 4 |
Trade off
Comparison matrix
From Frame anatomy: the ebp+8 rule: every row here is a choice with a cost. Fill the Offset from ebp column, then say which row you would actually pick and what you give up for it.
| Slot | Offset from ebp |
|---|---|
| first argument | ebp + 8 |
| saved return addr (rip) | ebp + 4 |
| saved frame ptr (sfp) | ebp + 0 |
| first local | ebp − 4 |
Anomaly
Predict first
A student writes this, and it looks reasonable:
Read 4(%ebp) as the first argument
It is wrong. Say what breaks — and say it before you turn the page.
Correct: Forgets that the rip sits between the saved ebp and the arguments — ebp+4 is the return address, not arg1.
Finding arg1 from ebp.
Why: Forgets that the rip sits between the saved ebp and the arguments — ebp+4 is the return address, not arg1.
Trap
Finding arg1 from ebp.
Read 4(%ebp) as the first argument
Why: Forgets that the rip sits between the saved ebp and the arguments — ebp+4 is the return address, not arg1.
Finding arg1 from ebp.
Read 8(%ebp) as the first argument
Why: §2.8: ebp+0 = sfp, ebp+4 = rip, ebp+8 = arg1. Two saved words sit between ebp and the arguments. (This same +4-to-the-rip fact powers the format-string attack in Lesson 9.)
Section
Part 3 · §2.9
Pattern
Predict first
The table runs: push $2 / push $1 | 1 | args on stack, reverse order · call foo | 2–3 | push rip; eip = &foo · push %ebp | 4 | save sfp · mov %esp,%ebp | 5 | ebp = esp · sub $16,%esp | 6 | locals · leave | 8–9 | esp=ebp; pop ebp
In The textbook's foo(1,2), annotated, given the rows so far: what is the next one — the row where Instruction is ret?
Correct: ret | 10 | pop rip into eip
| Instruction | Step | Effect |
|---|---|---|
| push $2 / push $1 | 1 | args on stack, reverse order |
| call foo | 2–3 | push rip; eip = &foo |
| push %ebp | 4 | save sfp |
| mov %esp,%ebp | 5 | ebp = esp |
| sub $16,%esp | 6 | locals |
| leave | 8–9 | esp=ebp; pop ebp |
| ret | 10 | pop rip into eip |
Why: The relationship between the columns, not the individual numbers, is what generates the next row. push %ebp ; mov %esp,%ebp ; sub $16,%esp appears at the start of essentially every -O0 C function.
Worked example
main:
push $2 # step 1: arg2
push $1 # step 1: arg1
call foo # steps 2-3: push rip, eip = &foo
add $8, %esp # step 11: remove arguments
foo:
push %ebp # step 4: save sfp
mov %esp, %ebp # step 5: anchor frame
sub $16, %esp # step 6: allocate locals
# ... step 7: body ...
mov %ebp, %esp # step 8
pop %ebp # step 9 (8+9 = leave)
ret # step 10 (pop rip -> eip)Prologue = steps 4–6
Why: push %ebp ; mov %esp,%ebp ; sub $16,%esp appears at the start of essentially every -O0 C function.
Epilogue = steps 8–10
Why: mov %ebp,%esp ; pop %ebp is abbreviated leave; the final ret pops the rip into eip. The 16 in sub $16 is the compiler's chosen local size.
| Instruction | Step | Effect |
|---|---|---|
| push $2 / push $1 | 1 | args on stack, reverse order |
| call foo | 2–3 | push rip; eip = &foo |
| push %ebp | 4 | save sfp |
| mov %esp,%ebp | 5 | ebp = esp |
| sub $16,%esp | 6 | locals |
| leave | 8–9 | esp=ebp; pop ebp |
| ret | 10 | pop rip into eip |
Comparison
Comparison matrix
From The textbook's foo(1,2), annotated: refill the Effect column from what you know. The rest of the table is as it appeared.
| Instruction | Step | Effect |
|---|---|---|
| push $2 / push $1 | 1 | args on stack, reverse order |
| call foo | 2–3 | push rip; eip = &foo |
| push %ebp | 4 | save sfp |
| mov %esp,%ebp | 5 | ebp = esp |
| sub $16,%esp | 6 | locals |
| leave | 8–9 | esp=ebp; pop ebp |
| ret | 10 | pop rip into eip |
Anomaly
Predict first
A student writes this, and it looks reasonable:
Treat ret as a jump to a label the compiler hard-coded
It is wrong. Say what breaks — and say it before you turn the page.
Correct: Misses that ret reads its target FROM THE STACK — whatever 4 bytes sit at esp.
Misses that ret reads its target FROM THE STACK — whatever 4 bytes sit at esp.
Why: Misses that ret reads its target FROM THE STACK — whatever 4 bytes sit at esp.
Trap
What does ret do?
Treat ret as a jump to a label the compiler hard-coded
Why: Misses that ret reads its target FROM THE STACK — whatever 4 bytes sit at esp.
What does ret do?
ret pops the top-of-stack word into eip and resumes there
Why: §2.9: ret = pop %eip. The destination is data on the stack (the rip). Overwrite that data and ret carries you anywhere — that's the hijack primitive of Lesson 8.
Two truths and a lie
Sort into buckets
Some of these hold up and some are the exact mistakes this lesson is built to prevent. Sort them.
Constraint
Discussion prompt
Run The 11 steps at a glance with this step confiscated:
Execute body (args at ebp+8, locals below ebp).
Is it still possible? If it is, say what takes its place and what it costs you. If it is not, say exactly what that step was providing that nothing else does.
Hint: A step you can drop for free was never load-bearing. If you cannot drop it, name the thing that goes wrong the moment it is gone.
Answer:
Pattern
Edge cases
Discussion prompt
The 11 steps at a glance works on the cases you have just seen. Push it to the edge: what is the most degenerate input it still handles — empty, zero, one item, everything equal — and what is the first case where it stops being true? Name the case, not just "it breaks".
Hint: Try the smallest legal input, then the largest, then the one where two things collide. Methods are specified at their edges; the middle takes care of itself.
Answer:
Elimination
Eliminate the wrong options
At what offset from ebp is the saved return address (rip), and what is at ebp+8?
3 of these 4 are wrong. Strike them one at a time, and say what rules each one out before you strike the next. The survivor is the answer.
Survives elimination: A
Why: §2.8 frame layout: ebp+0 = saved frame pointer (sfp), ebp+4 = saved return address (rip), ebp+8 = first argument. Locals are below ebp at negative offsets.
Check
ebp is anchored for foo. You need the saved return address. Solve on paper.
Check your understanding
At what offset from ebp is the saved return address (rip), and what is at ebp+8?
Answer: A
Why: §2.8 frame layout: ebp+0 = saved frame pointer (sfp), ebp+4 = saved return address (rip), ebp+8 = first argument. Locals are below ebp at negative offsets.
Elimination
Eliminate the wrong options
By how many bytes has esp moved (net), and in which direction?
3 of these 4 are wrong. Strike them one at a time, and say what rules each one out before you strike the next. The survivor is the answer.
Survives elimination: A
Why: Each push lowers esp by 4: two args (−8), the call pushes the rip (−4), the prologue pushes the sfp (−4) and then sub $16 (−16). Total −32, downward (the stack grows toward lower addresses).
Check
foo takes two 4-byte args and its prologue does sub $16, %esp. Count from just before the args are pushed to just after the prologue finishes.
Check your understanding
By how many bytes has esp moved (net), and in which direction?
Answer: A
Why: Each push lowers esp by 4: two args (−8), the call pushes the rip (−4), the prologue pushes the sfp (−4) and then sub $16 (−16). Total −32, downward (the stack grows toward lower addresses).
Check
The call convention explicitly saves the old eip (as rip) and old ebp (as sfp), but never saves the old esp.
Check your understanding
Why is saving esp unnecessary?
Answer: B
Why: Every byte esp is decremented on the way in (args, rip, sfp, locals) is matched by an increment on the way out (leave, ret, arg cleanup). esp self-restores to exactly its pre-call value, so there is nothing to save.
Concept
mov %ebp,%esp ; pop %ebp (steps 8–9).add $8,%esp) runs back in main.Sorting
Sort into buckets
These are the pieces of L06 · The 11-Step Calling Convention & Stack-Frame Anatomy, out of order. Put each one back under the part of the lesson it belongs to.
Concept
sub $N is the local space a buffer overflows out of (Lesson 8).Concept
gcc -m32 -O0 -S foo.c and match each emitted line to one of the 11 steps; then single-step with gdb stepi watching $esp, $ebp, $eip.Connect it up
Draw it
One page, no notation unless you need it: draw how these connect — The Calling Convention · The 11 Steps · In Assembly. Put an arrow wherever one of them is what makes another possible, and label the arrow with why.
Recap
You can run the 11 steps, point to the prologue and epilogue in assembly, label a frame high-to-low, find arg1 at ebp+8, and explain esp's self-restoring balance.
| Slot (high→low) | Offset | Set by |
|---|---|---|
| arg2 / arg1 | ebp+12 / +8 | caller push (step 1) |
| rip | ebp+4 | call (step 2) |
| sfp | ebp+0 | prologue (step 4) |
| locals | ebp−4 … esp | prologue (step 6) |
Want this taught 1-on-1? Alexander tutors Computer Security — $55/session, free consultation.