Opens the hash-function black box: how Merkle-Damgård pads, splits and chains a fixed-size state, MD5 traced by hand on real values (padding, message words, one operation, the feed-forward that makes it one-way), how chosen-prefix collisions broke MD5 and SHA-1 for signatures while preimages still hold, and why SHA-3 came from a public competition with a sponge design.
Subject: Computer Security · 68 slides · code lesson
Open the interactive version of this deck
Title
Computer Security — Crypto Lecture 2
How a fixed-size box hashes a file of any size, what MD5 does to every block, and how MD5 and SHA-1 were broken
Section
Four names, and where each one stands today
Objectives
The last lesson treated a hash function as a black box with three promises. This one opens the box. By the end you should be able to trace MD5 by hand on a real input, explain why its design looks the way it does, and say precisely what broke in MD5 and SHA-1 and what did not.
Security I, Lecture 2: Cryptography, part one (hash functions) part one: hash functions
Warm-up
Before looking at real hash functions, recall the three properties from the last lesson, because broken always refers to one of them.
Discussion prompt
Name the three security properties of a hash function, in order of how hard they are to break by brute force for a 128-bit digest. Which one would you expect to fall first, and why?
Hint: Remember the birthday bound: half the digest length.
Answer:
One-wayness (find any input for a given digest) and second-preimage resistance (find a different input with the same digest as a given one) both cost about 2 to the 128 tries by brute force. Collision resistance (find any two inputs with the same digest) costs only about 2 to the 64, by the birthday bound.
So collision resistance is the weakest point by design, and it is exactly the property that fell for MD5 and SHA-1. When someone says MD5 is broken, they mean collisions can be found. Nobody can reverse MD5.
Concept
Four families cover almost everything you will meet. Two are broken for collision resistance and must never protect anything an attacker could target; two are fine.
| function | year | digest | status today |
|---|---|---|---|
| MD5 | 1992 | 128 bits | Broken: a laptop finds a collision in seconds. Never use it for security. |
| SHA-1 | 1995 | 160 bits | Broken: first real collision 2017, chosen-prefix collisions 2020. Deprecated everywhere. |
| SHA-256, SHA-512 | 2001 | 256, 512 bits | Fine. The SHA-2 family, used in TLS, Git's newer object format, Bitcoin and package managers. |
| SHA-3 | 2015 | 224 to 512 bits | Fine. A completely different design, kept as a backup in case SHA-2 falls. |
Read broken precisely. For MD5 and SHA-1 it means collisions are findable. Preimage resistance still holds for both: nobody can take an MD5 digest and produce an input for it. That is why an old HMAC-MD5 is not an emergency, while a certificate signed over an MD5 digest is.
Security I, Lecture 2: Cryptography, part one (hash functions) slide s-common-hash
Discrimination
Collision resistance is gone; preimage resistance and accidental-error detection remain. Sort each use by whether a broken collision property actually hurts it.
Sort into buckets
Does this use depend on collision resistance against an attacker?
Section
How a fixed-size box hashes a file of any size
Socratic
MD5's core is a box with a fixed-size input and a fixed-size output. Yet md5sum happily hashes a 5-byte string or a 6 GB disk image.
Discussion prompt
Sketch a way to use one fixed-size mixing step to fingerprint input of any length. What do you need to keep between steps?
Hint: Think of reading a book one page at a time while keeping a single note.
Answer:
Cut the input into fixed-size pieces, keep one running value, and fold the pieces in one at a time. The running value after the last piece is the fingerprint. The only thing carried between steps is that running value, called the state.
This is the whole architecture of MD5, SHA-1 and SHA-256. Everything else in this lesson is about making the folding step good, not about changing the shape.
Worked example
Make the idea concrete with the weakest possible folding step: addition. Read the 12-byte file hello world! four bytes at a time, treat each chunk as a 32-bit number, and keep a running sum.
state = 0x00000000 # a fixed start value
state = state + 0x6c6c6568 # "hell" -> 6c6c6568
state = state + 0x6f77206f # "o wo" -> dbe385d7
state = state + 0x21646c72 # "rld!" -> fd47f249 the digest
# every + is mod 2^32: only the low 32 bits are kept| step | chunk | as a number | state after |
|---|---|---|---|
| start | 00000000 | ||
| 1 | hell | 6c6c6568 | 6c6c6568 |
| 2 | o wo | 6f77206f | dbe385d7 |
| 3 | rld! | 21646c72 | fd47f249 (the digest) |
Why 6c6c6568 for hell?
Why: The bytes are 68 65 6c 6c, and the chunk is read back to front (little-endian), the same convention MD5 uses.
Verify: Check step 2
Why: 6c6c6568 plus 6f77206f is dbe385d7, which fits in 32 bits, so no carry is dropped.
Picture it
The same computation as a picture. Hold on to this shape: fixed chunks, a running state, one mixing step per chunk, and the final state is the digest.
Figure (svg): Three chunks hell, o wo, rld! each feed a plus box; the running state goes from 00000000 to 6c6c6568 to dbe385d7 to the digest fd47f249
MD5 keeps exactly this picture and changes two things: the state grows from 32 bits to 128 bits, and the plus box becomes a 64-operation mixing function. The next part opens that function.
Security I, Lecture 2: Cryptography, part one (hash functions) slide s-md-why
Counterexample
The running sum has the right shape and a terrible folding step.
Discussion prompt
Without any computation, give a second 12-byte file with the same toy digest fd47f249 as hello world!. What property of addition did you use?
Hint: Does a + b + c change if you reorder the terms?
Answer:
Swap any two chunks: o wohellrld! also sums to fd47f249, because addition does not care about order. Collisions are everywhere, and they need no search at all.
This is why a real compress function must be order-sensitive and must mix every bit of a chunk into every bit of the state. A checksum like this catches random errors reasonably well, but it is useless against anyone who chooses the input.
Concept
Now replace the toy's pieces with MD5's. There are three steps, and the same three steps run SHA-1 and SHA-256. The shape is named Merkle-Damgård after its two inventors.
Figure (svg): A 100-byte message padded with a 1 bit, 159 zero bits and the 64-bit length 800 to make 1024 bits; split into two 512-bit blocks; chained through two compress boxes starting from the IV
Two things remain unexplained: what the state looks like, and what the compress box does. The rest of the lesson fills in exactly those two.
Security I, Lecture 2: Cryptography, part one (hash functions) slide s-md
Fill the middle
Work through the padding for a 100-byte message. Blocks are 512 bits, and the length field always fills the last 64 bits of the last block.
Fill in the blanks
message length = 100 bytes = 800 bits
add one 1 bit, then zeros, then a 64-bit length field
smallest multiple of 512 that fits 800 + 1 + 64 bits = 1024 bits
number of zero bits = 1024 - 800 - 1 - 64 = 159
number of 512-bit blocks = 2
Why: 800 bits of message plus the 1 bit plus the 64-bit length is 865 bits, which does not fit in 512, so the padded message is 1024 bits. The zeros fill the gap: 1024 - 800 - 1 - 64 = 159. That is two blocks of 512 bits. A reference MD5 implementation produces exactly 128 bytes, two blocks, for a 100-byte input.
Prediction
The padding needs at least 1 bit plus the 64-bit length, so at least 9 bytes once you round the 1 bit up to the byte 0x80. One block holds 64 bytes.
Predict first
How many 512-bit blocks does MD5 process for a 55-byte message, and for a 56-byte one?
Correct: 1 and 2
Why: 55 bytes plus the 0x80 byte plus 8 length bytes is exactly 64: one block. At 56 bytes there is no room left for the length field after the 0x80 byte, so the padding spills into a second block that is almost all zeros. A reference implementation confirms it: 55 bytes give 1 block; 56, 63 and 64 bytes all give 2.
Socratic
The padding could have been just a 1 bit and zeros. MD5 also writes the message length, always in the last 8 bytes.
Discussion prompt
Suppose the padding were only zero bytes. What happens to the messages abc and abc followed by one zero byte? Then: what do the 1 bit and the length field each add?
Hint: Two different messages must never pad to the same bits.
Answer:
With zeros alone, abc and abc plus a zero byte pad to exactly the same bits, so they collide for free. The single 1 bit fixes that by marking where the message stops: whatever comes after it is padding.
The length in a fixed place, the last 8 bytes, goes further: two messages of different lengths always differ in their final block. This length strengthening is what lets the Merkle-Damgård proof say that any collision in the full hash forces a collision inside one compress box, so a good box gives a good hash.
Estimation
Each block is 64 bytes, and every block goes through the same compress box with the state left by the previous one.
Predict first
Roughly how many compress calls does md5sum make on a 6,000,000,000-byte file?
Correct: about 94 million
Why: 6,000,000,000 divided by 64 is 93,750,000, so about 94 million blocks, plus at most one for padding. The state carries everything seen so far between them, which is why hashing is a single pass and needs only 16 bytes of state no matter how large the file is.
Section
The state, the block, the schedule and two fixed tables
Concept
MD5's state is 128 bits, held as four 32-bit words. In 1991 a 32-bit word was one CPU register, so every change to the state is a handful of machine instructions.
Figure (svg): Four registers A, B, C, D with their starting constants 67452301, efcdab89, 98badcfe, 10325476 and their final values for hello, which spell the digest 5d41402a bc4b2a76 b9719d91 1017c592
$ echo -n "hello" | md5sum
5d41402abc4b2a76b9719d911017c592 -
# A = 5d41402a B = bc4b2a76 C = b9719d91 D = 1017c592| word | starts at | after hello (as printed) |
|---|---|---|
| A | 67452301 | 5d41402a |
| B | efcdab89 | bc4b2a76 |
| C | 98badcfe | b9719d91 |
| D | 10325476 | 1017c592 |
The state starts at four constants printed in the standard, is changed 64 times per block, and whatever it holds after the last block is written out as the digest. Each change rewrites exactly one word, using all four words, one word of the message and one constant.
Security I, Lecture 2: Cryptography, part one (hash functions) slide s-md5-state
Notation
The digest is nothing more than the final state printed in order. Open the notes to see how the 32 hex digits map onto the registers.
Annotate
Remember the last note. Because the digest is the complete state, MD5's output is also a valid starting point for further hashing. The MAC lesson explains why that matters for keyed hashing.
Security I, Lecture 2: Cryptography, part one (hash functions) slide s-md5-state
Concept
The other input to the compress box is one 512-bit block, read as 16 words of 32 bits called M[0] to M[15]. Here is the single block for hello, padded.
"hello" = 68 65 6c 6c 6f # 5 bytes = 40 bits
padded = 68 65 6c 6c 6f 80 00 00 ... 00 # 80 = the single 1 bit, then zeros
28 00 00 00 00 00 00 00 # last 8 bytes: the length 40 = 0x28
M[0] = 6c6c6568 # bytes 68 65 6c 6c, read back to front
M[1] = 0000806f # bytes 6f 80 00 00
M[2] .. M[13] = 00000000
M[14] = 00000028 # the length
M[15] = 00000000| word | bytes | value |
|---|---|---|
| M[0] | 68 65 6c 6c (hell) | 6c6c6568 |
| M[1] | 6f 80 00 00 (o, then the 1 bit) | 0000806f |
| M[2] to M[13] | all zero | 00000000 |
| M[14] | 28 00 00 00 (length 40) | 00000028 |
| M[15] | 00 00 00 00 (top half of the length) | 00000000 |
Each of the 64 operations consumes one word, so with 16 words every word is used four times, once in each round. The bytes of a word are read back to front, which is why hell becomes 6c6c6568: a convention of the standard, not a security feature.
Security I, Lecture 2: Cryptography, part one (hash functions) slide s-md5-block
Faded example
The message is hi!, three bytes: 68 69 21. Pad it the MD5 way and read the words. The first word is done for you.
Fill in the blanks
padded bytes : 68 69 21 80 00 00 ... 00 | 18 00 00 00 00 00 00 00
M[0] = 80216968 # bytes 68 69 21 80, back to front
M[1] = 00000000
M[14] = 00000018 # the length in bits, 3 x 8
M[15] = 00000000
Why: Three bytes plus the 0x80 byte fill exactly M[0], so M[1] through M[13] are zero. The length is 3 bytes = 24 bits = 0x18, written little-endian in the last 8 bytes, so M[14] = 00000018 and M[15] = 00000000. A reference MD5 implementation reads exactly these words from the padded block.
Picture it
The same block as a picture. Only three of the sixteen words are non-zero for a short message.
Figure (svg): Sixteen boxes M[0] to M[15]; M[0] = 6c6c6568, M[1] = 0000806f and M[14] = 00000028 are highlighted, the rest are zero
For a long message the early blocks are pure message, and only the last block carries the 0x80 byte and the length. Every block, padded or not, goes through the identical compress box.
Security I, Lecture 2: Cryptography, part one (hash functions) slide s-md5-block
Concept
Inside the compress box are 64 operations in a row, grouped into four rounds of 16. Every block runs the same 64 operations in the same order. Each operation reads one of the block's 16 words, and a fixed schedule says which.
Figure (svg): A grid of four rounds by sixteen operations showing which message word each operation reads: round 1 in order, round 2 by 5j+1, round 3 by 3j+5, round 4 by 7j, all mod 16; M[0] is highlighted once per row
Read one row: round 1 uses M[0], M[1], up to M[15] in order; the other rounds shuffle the same 16 words. Every word is used exactly once per round. So the recipe never changes, only the ingredients: block 2 runs the same 64 operations with its own 16 words and with the state block 1 left behind.
Security I, Lecture 2: Cryptography, part one (hash functions) slide s-md5-zoom
Pattern
Round 2 reads the words in this order: 1, 6, 11, 0, 5, 10, 15, 4, 9, 14, 3, 8, ... Each step adds the same amount, wrapping around at 16.
Predict first
Which word does the 13th operation of round 2 (j = 12) read?
Correct: M[13]
Why: The rule is k = (5j + 1) mod 16. For j = 12 that is 61 mod 16 = 13. Continuing the sequence by adding 5 each time and wrapping also gives 8 + 5 = 13. The full row is 1, 6, 11, 0, 5, 10, 15, 4, 9, 14, 3, 8, 13, 2, 7, 12.
Security I, Lecture 2: Cryptography, part one (hash functions) slide s-md5-zoom
Fill the middle
Round 3 reads word (3j + 5) mod 16 and round 4 reads word 7j mod 16, where j counts 0 to 15 within the round.
Fill in the blanks
round 3, j = 0 : (3 x 0 + 5) mod 16 = 5
round 3, j = 4 : (3 x 4 + 5) mod 16 = 1
round 4, j = 1 : 7 x 1 mod 16 = 7
round 4, j = 3 : 7 x 3 mod 16 = 5
Why: Round 3: j = 0 gives 5, and j = 4 gives 17 mod 16 = 1. Round 4: j = 1 gives 7, and j = 3 gives 21 mod 16 = 5. Because 5, 3 and 7 are all odd, each rule hits every value 0 to 15 exactly once, which is why every word is read once per round.
Concept
An operation needs two more inputs, and both are fixed tables, the same for every block of every file. K[i] is a 32-bit constant added in at operation i. s[i] is how many bit positions operation i rotates by.
import math
K = [int(abs(math.sin(i + 1)) * 2**32) & 0xffffffff for i in range(64)]
# K[0] = d76aa478 K[1] = e8c7b756 K[2] = 242070db K[3] = c1bdceee ... K[63] = eb86d391
s = [7, 12, 17, 22] * 4 + [5, 9, 14, 20] * 4 + [4, 11, 16, 23] * 4 + [6, 10, 15, 21] * 4| table | entries | where they come from | job |
|---|---|---|---|
| K[i] | 64, one per operation | the sine function, scaled to 32 bits | makes every operation different |
| s[i] | four values per round, repeated | chosen by the designer | moves bits to new positions |
| round 1 s values | 7, 12, 17, 22 | repeated four times | |
| round 4 s values | 6, 10, 15, 21 | repeated four times |
Rotating left by s[i] slides every bit of the running sum s[i] places up, so a bit that sat low in one operation sits high in the next. Each round cycles through its own four rotation amounts. Running the Python above reproduces all 64 constants of the standard.
Security I, Lecture 2: Cryptography, part one (hash functions) slide s-md5-ks
Socratic
The designer could have picked any 64 random-looking numbers for K. Instead they are the first 32 bits of the fractional sine values, which anyone can recompute.
Discussion prompt
What would users worry about if the 64 constants had been typed in with no explanation, and how does deriving them from sine answer that worry?
Hint: Who could have picked the numbers, and how would you know they picked fairly?
Answer:
Unexplained constants raise the question of whether the designer chose them to plant a hidden weakness that only they know how to use. Constants that come from a simple public formula leave no room to choose: anyone can check them, and nobody could have tuned them toward a trapdoor.
These are called nothing-up-my-sleeve numbers. The contrast case is Dual_EC_DRBG, a random-number standard whose unexplained constants were later widely believed to hide exactly such a trapdoor. The same instinct, public and explainable design, is why SHA-3 was chosen by open competition, as Part 7 shows.
Section
Three tools, wired together, run 64 times per block
Concept
Every MD5 operation is built from just three tools. Here they are on 8-bit words so the bits fit on a slide; MD5 does the same on 32-bit words.
| tool | inputs | 8-bit example | result |
|---|---|---|---|
| mix F | B, C, D | B = 11001010, C = 10101010, D = 01010101 | 10011111 |
| add, keep 8 bits | two words | 11110000 + 00110000 = 100100000 | 00100000 (carry dropped) |
| rotate left 3 | one word | 10110001 | 10001101 (bits wrap around) |
mix F — a bitwise choice: wherever B has a 1, take C's bit; wherever B has a 0, take D's bit. Rounds 2 to 4 swap F for the mixes G, H and I
add mod 2 to the 32 — ordinary addition that throws away any carry out of the top bit, so the result stays one word; this is where the message word M[i] and the constant K[i] enter
rotate left by s[i] — shift every bit up s[i] places; bits that fall off the top come back in at the bottom
That is all an operation needs: one mix, some additions, one rotation. The mix is the only part that changes from round to round.
Security I, Lecture 2: Cryptography, part one (hash functions) slide s-md5-tools
Faded example
Same tools, new 8-bit inputs. Work bit by bit. For F: where B is 1 take C, where B is 0 take D.
Fill in the blanks
F(B=11110000, C=10101010, D=01010101) = 10100101
11000000 + 01000001, keep 8 bits = 00000001
rotate 10010110 left by 2 = 01011010
Why: F: the top four bits of B are 1, so take C's top four (1010); the bottom four are 0, so take D's bottom four (0101), giving 10100101. Add: 192 + 65 = 257 = 1 0000 0001, and dropping the ninth bit leaves 00000001. Rotate: shifting 10010110 up two places pushes out 10, which wraps to the bottom: 01011010.
Concept
Now wire the three tools together. Three registers simply slide one place over, and the new B is cooked from all four registers, one message word, and one constant, with a rotation in the middle.
Figure (svg): Registers A B C D at the top and new A B C D at the bottom: old D becomes new A, old B becomes new C, old C becomes new D; the new B is A plus mix(B,C,D) plus M[i] plus K[i], rotated left by s[i], plus old B
new_B = B + rotl( A + mix(B, C, D) + M[i] + K[i], s[i] ) # every + is mod 2^32
new_A, new_C, new_D = D, B, C # the other three slide over
# mix = F in round 1, then G, H, I in rounds 2, 3, 4| register | comes from |
|---|---|
| new A | old D |
| new B | B + rotl(A + mix + M[i] + K[i], s[i]) |
| new C | old B |
| new D | old C |
Every plus is addition modulo 2 to the 32. Sixty-four of these operations absorb one block. After four operations every register has been rewritten once.
Security I, Lecture 2: Cryptography, part one (hash functions) slide s-md5-op
Notation
This one line is the heart of MD5. Open each note to see what each piece contributes.
Annotate
Read it as: mix the state, add the message and a constant, scramble the positions, add back. Repeat 64 times with different settings.
Security I, Lecture 2: Cryptography, part one (hash functions) slide s-md5-op
Worked example
Now real numbers. This is the very first operation MD5 performs when hashing hello: round 1, so the mix is F, the word is M[0], the constant is K[0] = d76aa478 and the rotation is 7. Every value was produced by a from-scratch MD5 that matches Python's hashlib.
# MD5 on "hello", block 1, operation 0 (round 1: mix F, word M[0], K[0], s = 7)
A, B, C, D = 67452301, efcdab89, 98badcfe, 10325476 # the IV
F(B, C, D) = 98badcfe
A + F = ffffffff
+ M[0] (6c6c6568) = 6c6c6567 # the carry out of bit 31 is dropped
+ K[0] (d76aa478) = 43d709df
rotl(43d709df, 7) = eb84efa1
new_B = B + eb84efa1 = db529b2a
new A, C, D = 10325476, efcdab89, 98badcfe| step | value |
|---|---|
| F(B, C, D) | 98badcfe |
| A + F | ffffffff |
| + M[0] | 6c6c6567 |
| + K[0] | 43d709df |
| rotate left 7 | eb84efa1 |
| + old B = new B | db529b2a |
Why F(B, C, D) = C here
Why: With the IV, B = efcdab89 and C = 98badcfe, and the choice happens to select C's bits everywhere they differ from D's, so F is 98badcfe.
Watch the carry
Why: ffffffff + 6c6c6568 overflows 32 bits; dropping the carry leaves 6c6c6567.
Verify: The register slide
Why: new A = old D = 10325476, new C = old B = efcdab89, new D = old C = 98badcfe, which is the state operation 1 starts from in the reference trace.
Picture it
Map the worked example back onto the wiring diagram: the adder chain produced 43d709df, the rotation produced eb84efa1, and adding old B gave the new B.
Figure (svg): The MD5 operation diagram again: mix, adder chain, rotation and final add feeding the new B, with the other three registers sliding over
Operation 1 now runs with A = 10325476, B = db529b2a, C = efcdab89, D = 98badcfe, the word M[1] = 0000806f, K[1] = e8c7b756 and a rotation of 12. The reference trace gives its new B as 11141778.
Security I, Lecture 2: Cryptography, part one (hash functions) slide s-md5-op
Prediction
After operation 0 the state is A = 10325476, B = db529b2a, C = efcdab89, D = 98badcfe. Operation 1 computes a new B of 11141778.
Predict first
What is the full state after operation 1?
Correct: A = 98badcfe, B = 11141778, C = db529b2a, D = efcdab89
Why: Only B is computed; the rest slide: new A = old D = 98badcfe, new C = old B = db529b2a, new D = old C = efcdab89. This matches the reference trace, where operation 2 starts from exactly these four values.
Section
Three settings change; the wiring never does
Concept
All 64 operations share the wiring you just traced. Only three settings change with the round: which mix, the order the words are read in, and the rotation amounts. K[i] also differs in every operation.
| round | mix of B, C, D | in words | word order | rotations |
|---|---|---|---|---|
| 1 (ops 0-15) | F = (B and C) or (not B and D) | choose, selected by B | j | 7, 12, 17, 22 |
| 2 (ops 16-31) | G = (B and D) or (C and not D) | choose, selected by D | 5j + 1 | 5, 9, 14, 20 |
| 3 (ops 32-47) | H = B xor C xor D | parity | 3j + 5 | 4, 11, 16, 23 |
| 4 (ops 48-63) | I = C xor (B or not D) | parity with an OR inside | 7j | 6, 10, 15, 21 |
You do not need to memorise any of these numbers. The point is the design pattern: the same operation, deliberately varied in function, order and rotation, so that no structure from one round lines up with the next.
Security I, Lecture 2: Cryptography, part one (hash functions) slide s-md5-rounds
Matching
Each round's mix combines B, C and D in its own way.
Match the pairs
Why: F and G are both choose functions; F lets B select between C and D, while G lets D select between B and C. H is plain parity. I mixes an OR into a parity, so it is not linear like H. Varying the function per round means a pattern that passes through one round cleanly is disturbed by the next.
Two truths and a lie
Each statement is about the structure inside MD5's compress box.
Eliminate the wrong options
Which statement is the myth?
Survives elimination: c
Why: The myth is c. Every block of every file runs the identical 64 operations with the same constants and rotations. Only the 16 words and the incoming state differ from block to block. That fixed recipe is what makes MD5 easy to implement in a few dozen lines.
Concept
Assemble the pieces. The state enters, four rounds of sixteen operations rewrite it, the block's words feed in once per round, and finally the input state is added back, word by word.
Figure (svg): State in, four round boxes with their mixes, word orders and rotations, then a feed-forward that adds the input state back, giving the state out
That last step is called the feed-forward, and it has a precise job. Every one of the 64 operations can be undone if you know its output and its inputs M[i], K[i] and s[i], all of which are public or part of the block. Without the feed-forward, the whole box could be run backwards. Adding the input state back is what makes the box one-way.
Security I, Lecture 2: Cryptography, part one (hash functions) slide s-md5
Explain it to yourself
Here is operation 0 of the hello block run backwards, using only its output state and the public values M[0], K[0] and s[0] = 7.
# given the state after operation 0, and the public M[0], K[0], s[0]:
old_B, old_C, old_D = new_C, new_D, new_A = efcdab89, 98badcfe, 10325476
diff = new_B - old_B = eb84efa1
sum = rotr(diff, 7) = 43d709df
old_A = sum - F(old_B, old_C, old_D) - M[0] - K[0] = 67452301 # the IV again| line | value recovered |
|---|---|
| old B, C, D | efcdab89, 98badcfe, 10325476 |
| diff | eb84efa1 |
| sum | 43d709df |
| old A | 67452301 |
Discussion prompt
Explain why each line works. Then say why this does NOT let anyone reverse a full MD5 digest.
Hint: Which value would you need to subtract back out of the digest?
Answer:
Three registers only slid over, so old B, C and D are read straight off the new state. Subtracting old B undoes the final add; rotating right by 7 undoes the rotate left by 7; subtracting the mix, the word and the constant undoes the adder chain. Each operation is a permutation of the state, so all 64 can be undone.
The digest, though, is the output after the feed-forward: the state that went in has been added to the result. To undo the rounds you would first have to subtract that input state, which is exactly what you do not know. That is the job of the feed-forward.
Security I, Lecture 2: Cryptography, part one (hash functions) slide s-md5
Faded example
After all 64 operations on the hello block the registers are A = c2fb1e5c, B = 865ca033, C = f8e294bb, D = 8292c29a. Add the IV back in, mod 2 to the 32. A is done for you.
Fill in the blanks
A = 67452301 + c2fb1e5c = 2a40415d
B = efcdab89 + 865ca033 = 762a4bbc
C = 98badcfe + f8e294bb = 919d71b9
D = 10325476 + 8292c29a = 92c51710
Why: B: efcdab89 + 865ca033 = 1762a4bbc, keep 32 bits: 762a4bbc. C: 98badcfe + f8e294bb = 1919d71b9, keep 919d71b9. D: 10325476 + 8292c29a = 92c51710, no carry. Printed little-endian these are 5d41402a bc4b2a76 b9719d91 1017c592, which is exactly md5sum's output for hello.
Worked example
Put the feed-forward and the output step together for the single-block message hello.
# after all 64 operations of the single "hello" block:
A, B, C, D = c2fb1e5c, 865ca033, f8e294bb, 8292c29a
# feed-forward: add the state that came in (the IV), word by word, mod 2^32
A = 67452301 + c2fb1e5c = 2a40415d -> printed little-endian: 5d41402a
B = efcdab89 + 865ca033 = 762a4bbc -> bc4b2a76
C = 98badcfe + f8e294bb = 919d71b9 -> b9719d91
D = 10325476 + 8292c29a = 92c51710 -> 1017c592| register | after 64 ops | plus IV | printed bytes |
|---|---|---|---|
| A | c2fb1e5c | 2a40415d | 5d 41 40 2a |
| B | 865ca033 | 762a4bbc | bc 4b 2a 76 |
| C | f8e294bb | 919d71b9 | b9 71 9d 91 |
| D | 8292c29a | 92c51710 | 10 17 c5 92 |
Why the bytes come out reversed
Why: MD5 stores each word little-endian, lowest byte first, and prints the bytes in memory order.
Verify: Check against md5sum
Why: Concatenating the printed bytes gives 5d41402abc4b2a76b9719d911017c592, which is what echo -n hello | md5sum prints and what Python's hashlib.md5 returns.
Hypothesis
Imagine a variant of MD5 that outputs the state straight after the 64 operations, with no add-back.
Predict first
For a one-block message, what could anyone do with such a digest?
Correct: Run the 64 operations backwards from the digest and the block, recovering the IV, so the box would not be one-way on its own
Why: Each operation is invertible given its public inputs, so the box without feed-forward is a permutation of the state for a fixed block. One-wayness of the compress function, which Merkle-Damgård relies on, would be lost. The feed-forward costs four additions and restores it.
Pattern
Everything about MD5, top to bottom, as one procedure.
Figure (svg): MD5 in one picture: the IV and a chain of compress boxes, one per block, ending in the digest; inside each box four rounds of sixteen operations and a feed-forward
The digest is the chain state after the last block. That one fact explains why MD5-style hashes are never used raw as MACs, and MD5 collisions can be found in seconds today, so MD5 must never be used for signatures.
Security I, Lecture 2: Cryptography, part one (hash functions) slide s-md5-recap
Check
Use the operation formula: new B = B + rotl(A + mix(B, C, D) + M[i] + K[i], s[i]), then A, C, D slide over.
Check your understanding
Before an operation the state is A = a, B = b, C = c, D = d. Which register ends up holding the old value of C?
Answer: D
Why: The slide is new A = old D, new C = old B, new D = old C. So the old value of C moves into D, and only B is computed fresh.
Check
Solve it on paper before you click.
Check your understanding
What is the job of adding the input state back at the end of MD5's compress box?
Answer: A
Why: Each operation is a permutation given public inputs. Adding the input state to the output means running the rounds backwards would require already knowing that input state, so the compress function becomes one-way.
Trap
A common belief: since MD5 is broken, anyone can take an MD5 digest and work out what was hashed.
| claim | reality |
|---|---|
| MD5 is broken | true, for collision resistance |
| so digests can be reversed | false: no practical preimage attack exists |
| so MD5 hides passwords fine | also false, for a different reason |
Broken means an attacker can construct two inputs with the same digest. Preimage resistance still holds for MD5 and SHA-1; the best known MD5 preimage attack costs about 2 to the 123 work.
Cracked MD5 password hashes come from guessing, not reversing: MD5 is so fast that a single GPU tries over a hundred billion guesses per second. That is why passwords need slow, salted hashes, a separate lesson. Keep the three properties separate whenever you read that a hash is broken.
Security I, Lecture 2: Cryptography, part one (hash functions) slide s-sha1-numbers
Section
From a curiosity to a weapon, twice
Concept
MD5's collision resistance did not fail all at once. Watch how each step made the weakness more useful to an attacker.
Figure (svg): Timeline: 2004 first collision of random-looking blocks; 2007 chosen-prefix collisions for any two documents; 2008 a rogue CA certificate; 2012 Flame malware faked a Microsoft code signature
| year | what happened | why it mattered |
|---|---|---|
| 2004 | first collision (Wang and colleagues): two messages differing in 128 random-looking bytes | proof the property was gone, but the inputs were garbage |
| 2007 | chosen-prefix collisions (Stevens, Lenstra, de Weger) | any two meaningful documents could be made to collide by appending junk tails |
| 2008 | a rogue CA certificate | one CA signature valid on two certificates; more in the PKI lesson |
| 2012 | Flame malware | an MD5 collision faked a Microsoft code signature that Windows Update accepted |
Eight years from academic curiosity to signed malware on Windows Update. In 2004 most people shrugged because the colliding messages were random. The lesson: retire a hash when the first crack appears.
Security I, Lecture 2: Cryptography, part one (hash functions) slide s-md5-hist
Ranking
Order the events from the first academic result to real-world exploitation.
Put in order
Why: 2004 first collision, 2007 chosen prefixes, 2008 rogue CA certificate, 2012 Flame. Each step needed the previous one: chosen prefixes turned random collisions into collisions between meaningful files, and that is what attacks on signatures require.
Socratic
A 2004-style collision gives two inputs that differ only in random-looking blocks. A chosen-prefix collision starts from any two documents the attacker picks and appends tails that make the digests match.
Discussion prompt
Conceptually, why does a signature over a digest fail once chosen-prefix collisions exist, while random collisions alone seemed harmless? What is the defence?
Hint: What exactly does a signer commit to: the document, or its digest?
Answer:
A signature vouches for a digest, so it vouches equally for every input with that digest. With chosen prefixes, two meaningful documents, one harmless and one malicious, can share a digest; getting the harmless one signed yields a valid signature on the other. Random collisions rarely gave anything meaningful to swap.
The defence is not cleverness at signing time but retiring the hash: sign with SHA-256 or better, and stop accepting signatures over MD5 or SHA-1. Certificate authorities also add unpredictable serial numbers, so the attacker cannot predict the exact bytes that will be signed.
Comparison
SHA-1, designed by the NSA and published by NIST in 1995, keeps MD5's shape and its step idea. Fill in the missing cells from what you know about MD5 and the hints in the row.
Comparison matrix
| MD5 | SHA-1 | |
|---|---|---|
| state | 128 bits, A B C D | 160 bits, A B C D E |
| shape | pad, 512-bit blocks, chain | the same, carrying 160 bits along |
| one block | 64 operations, 4 rounds of 16 | 80 steps, 4 rounds of 20 |
| mixes | F, G, H, I | choose, parity, majority, parity |
| constants | 64 values K[i], one per operation | 4 values, one per round |
| rotations | s[i] from a table | fixed: A by 5 and B by 30, every step |
| message words | 16 words, reused once per round | 16 words stretched to 80, one per step |
SHA-1 stretches its 16 words to 80 with XORs and a one-bit rotation, so every step gets its own word. Otherwise it is MD5's idea with a fifth register.
Security I, Lecture 2: Cryptography, part one (hash functions) slide s-sha1
Concept
Same design, same fate, thirteen years apart. The numbers explain why SHA-1 lasted longer.
| MD5 | SHA-1 | |
|---|---|---|
| digest | 128 bits, 32 hex digits | 160 bits, 40 hex digits |
| birthday bound | 2^64 hashes | 2^80 hashes |
| speed on one RTX 4090 | about 164 billion per second | about 51 billion per second |
| first real collision | 2004, Wang and colleagues | 2017, Google and CWI |
| collisions today | seconds on a laptop | the lecture estimates months on a few gaming GPUs |
| second preimage | no practical attack | no practical attack |
| retired | 2008 warnings; RFC 6151 in 2011 | NIST deprecated 2011; browsers reject it 2017 |
| still met in | file checksums, old password databases | Git object names, with SHA-256 the newer option |
Both still resist second preimages, which is why an old HMAC-MD5 is not an emergency while an MD5 or SHA-1 signature is. NIST plans to retire SHA-1 entirely by 2030. The GPU speeds are hashcat benchmarks quoted in the lecture.
Security I, Lecture 2: Cryptography, part one (hash functions) slide s-sha1-numbers
Estimation
A generic birthday attack on MD5 takes about 2 to the 64 hashes; on SHA-1 about 2 to the 80.
Predict first
How many times more work is the generic SHA-1 collision search?
Correct: 65,536 times
Why: 80 - 64 = 16 extra bits, and each bit doubles the work: 2 to the 16 = 65,536. SHA-1 is also about three times slower to compute than MD5, which adds a little more. That margin is why SHA-1's collisions came thirteen years after MD5's, once attacks found shortcuts below the generic bound.
Concept
Read the work factors on a scale where each step right doubles the work.
Figure (svg): A log scale from 2^0 to 2^160 marking 2^63 collisions after the 2005 attack, 2^80 the birthday bound by design, and 2^160 inversion by design
| work | meaning | status |
|---|---|---|
| 2^160 | invert SHA-1 by brute force | by design, still holds |
| 2^80 | a generic birthday collision | by design |
| about 2^63 | collisions after the 2005 attacks by Xiaoyun Wang and colleagues | the design cracked |
| 2017 | SHAttered: two PDFs with one SHA-1 digest | 6,500 CPU-years and 110 GPU-years, about 110,000 dollars of cloud time |
| 2020 | chosen-prefix collision (SHA-1 is a Shambles) | about 45,000 dollars |
By design every output bit depends on every input bit, so flipping one input bit flips about half the digest. The 2005 attacks steered carefully chosen bit differences through the 80 steps so they cancel out, far below the generic bound. Twelve years passed between theoretically weakened and here are two PDFs: that gap is the window in which everyone was supposed to migrate.
Security I, Lecture 2: Cryptography, part one (hash functions) slide s-sha1-strength
Stevens, Bursztein, Karpman, Albertini, Markov — The first collision for full SHA-1 (SHAttered, 2017) the SHAttered announcement, 2017
Estimation
The 2005 attack brought SHA-1 collisions from about 2 to the 80 work down to about 2 to the 63.
Predict first
Roughly how many times cheaper did that make a SHA-1 collision?
Correct: about 130,000 times
Why: 80 - 63 = 17, and 2 to the 17 = 131,072, so about 130,000 times cheaper. That moved the cost from far beyond any budget to a six-figure cloud bill, which is what Google and CWI eventually spent in 2017.
Discrimination
Use the distinction from this part: MD5 and SHA-1 are broken for collisions, not for preimages.
Sort into buckets
How urgent is each finding in a security review?
Section
A public competition, and a different design
Concept
After MD5 fell and SHA-1 cracked, NIST, the US standards body, did not design a replacement in-house. It ran an open contest.
| date | milestone |
|---|---|
| November 2007 | the call for submissions |
| October 2008 | 64 entries submitted; 51 accepted into round one |
| December 2010 | 5 finalists after two rounds |
| October 2012 | Keccak wins |
| August 2015 | SHA-3 published as FIPS 202 |
Anyone could submit a design, with the full specification and code published. For four years researchers everywhere tried to break every entry, and designs that cracked were dropped round by round. The same process produced AES and is producing the post-quantum standards.
Security I, Lecture 2: Cryptography, part one (hash functions) slide s-nist
NIST — SHA-3 Project (the hash function competition, 2007-2012) competition timeline
Explain it
A colleague argues that a secret, in-house design would be safer, because attackers could not study it.
Discussion prompt
Explain to them, in two or three sentences, why a public competition produces a more trustworthy hash function than a secret design.
Hint: Who gets to look for flaws in each case, and when?
Answer:
A hash function is only trusted after many capable people have tried and failed to break it. A public contest buys years of that attack effort before anything is deployed, instead of after, when a break is expensive to fix.
Secrecy does not stop attackers for long, since a deployed algorithm can be reverse-engineered, and it stops defenders from checking for weaknesses or trapdoors. This is Kerckhoffs's principle again. The contrast case is Dual_EC_DRBG, standardised without that open scrutiny and later abandoned.
Concept
The requirements explain the design: the same digest sizes as SHA-2 (224, 256, 384 and 512 bits) so it can drop in, at least the same security, public and royalty-free, and deliberately built unlike SHA-2.
Figure (svg): Top row: SHA-2 as a chain of compress boxes whose output is the whole final state, so it can be resumed. Bottom row: SHA-3 as a sponge that absorbs message blocks into the r part of a 1600-bit state, keeps the capacity c hidden, and squeezes the digest out of r
Practical advice from the lecture: use SHA-256 today. SHA-3 is the backup, built differently so that one break cannot take down both. MD5 and SHA-1 remain fine for spotting accidents, such as a download that broke halfway, but never for stopping an attacker.
Security I, Lecture 2: Cryptography, part one (hash functions) slide s-nist2
FIPS 202 — SHA-3 Standard: Permutation-Based Hash and Extendable-Output Functions (2015) section 4, the sponge construction
Trade off
Fill in the gaps using the status of each function and what each job actually needs.
Comparison matrix
| job | MD5 | SHA-256 | SHA-3-256 |
|---|---|---|---|
| detect a truncated download | fine | fine | fine |
| sign a software release | unsafe: collisions | fine | fine |
| used raw as a MAC, H(key || m) | unsafe | unsafe: length extension | no length extension, but use HMAC or KMAC |
| default choice today | no | yes | backup with a different design |
The row to remember is the third: the Merkle-Damgård shape you traced today is exactly why SHA-256 must not be used raw as a MAC, which is the starting point of the MAC lesson.
Real world
You use every one of these functions daily without seeing them.
Discussion prompt
For each, name where you have met it and whether that use is sound: (1) md5sum next to a Linux ISO, (2) a Git commit ID, (3) the certificate on an HTTPS site, (4) a Bitcoin block hash.
Hint: Ask in each case: could an attacker choose the inputs?
Answer:
(1) MD5 next to an ISO catches corrupted downloads; it is sound only if a signed SHA-256 sum or signature is also available for attackers. (2) Git object IDs are SHA-1; Git added collision detection after SHAttered and supports SHA-256 repositories as the newer option.
(3) HTTPS certificates are signed over SHA-256; browsers rejected SHA-1 certificates from 2017. (4) Bitcoin uses double SHA-256 for block hashes and proof of work. The sound uses all rely on SHA-2 wherever an attacker could choose inputs.
Check
Solve it on paper before you click.
Check your understanding
What is the main structural reason SHA-3 does not suffer from length extension, while SHA-256 does?
Answer: B
Why: A Merkle-Damgård digest is the full final state of the chain, so it can serve as a starting state. The sponge keeps the capacity part of its 1600-bit state hidden, so the output never gives away a complete state.
Connect it up
Connect everything from this lesson on one page.
Draw it
Draw MD5 from top to bottom: padding, blocks, the chained state, one compress box with four rounds and the feed-forward, and one operation inside it. Then add a branch for SHA-1 (what changed) and SHA-3 (what is different), and mark where collisions broke MD5 and SHA-1 and what still holds.
Check your drawing: the digest should be the last state; the feed-forward should sit at the end of the box; and the broken label should say collisions, not preimages.
Exit ticket
One question to check the central idea before you leave.
Predict first
A friend says: MD5's digest is just a summary computed at the end, unrelated to the internal state. Which correction is right?
Correct: The digest is the chain's final state after the last block's feed-forward, printed little-endian
Why: For hello you computed it yourself: the four registers after the feed-forward, 2a40415d, 762a4bbc, 919d71b9, 92c51710, printed byte-reversed, are the digest 5d41402abc4b2a76b9719d911017c592. Nothing is summarised separately.
Recap
You opened the black box. A hash function like MD5 is a short, fixed recipe run once per block, and every number on the way from hello to its digest is something you can now compute.
| idea | the one-line version |
|---|---|
| Merkle-Damgård | pad with 1, zeros and the length; chain a fixed state through one compress box per block |
| MD5 operation | new B = B + rotl(A + mix + M[k] + K[i], s[i]); A, C, D slide over |
| feed-forward | add the input state back, so the box cannot be run backwards |
| broken | collisions findable; preimages still hard |
| today | SHA-256 by default, SHA-3 as the backup, MD5 and SHA-1 only for accidents |
Security I, Lecture 2: Cryptography, part one (hash functions) part one: hash functions
Want this taught 1-on-1? Alexander tutors Computer Security — $55/session, free consultation.