Lesson 96: Graph Transformer & Graphormer

Match the pairs

From A transformer is a fully-connected graph — match each one to what it actually does. The descriptions have been shuffled.

• c1. Sequence input
• c2. Graph input
• c3. Key insight

• b1. Tokens are ordered 1-D; position encodes the 1-D index (sinusoidal or learned).
• b2. Nodes have arbitrary topology; structure encodes shortest-path distance, degree, and bond type.
• b3. Replace positional encoding with structural encoding — the attention computation stays identical.

Why: Sequence input, Graph input, Key insight are easy to tell apart while they are sitting next to their descriptions and much harder afterwards, which is what this checks.

Comparison matrix

From Toy molecule: 4-node graph: refill the Atom column from what you know. The rest of the table is as it appeared.

Sort into buckets

Sort these by deg, from memory, without looking back at Centrality encoding: degree as node identity. Telling them apart on the spot is the skill; the table is only where the answer happens to be written down.

2

C (0); N (1)

1

O (2); H (3)

g1

deg is "2" for C (0), N (1) — that is what the table on "Centrality encoding: degree as node…" records, and it is the single property separating this group from the rest.

g2

deg is "1" for O (2), H (3) — that is what the table on "Centrality encoding: degree as node…" records, and it is the single property separating this group from the rest.

Comparison matrix

From Spatial encoding: SPD → attention bias: every row here is a choice with a cost. Fill the phi(SPD) — toy learned value column, then say which row you would actually pick and what you give up for it.

Put in order

Put the moves of Worked example: biased attention for node 0 (C) into the order they have to happen.

1. Embed node features and compute raw Q·Kᵀ/√d_k
2. Add spatial bias B[0,j] = phi(SPD[0,j])
3. Softmax over biased logits to get attention weights for node 0

Why: These are the moves of the worked example in the order it makes them, and each one is set up by the one before it. Each node i is a 4-dim one-hot (atom type), linearly projected to d_model=8 then split into d_k=4 head queries/keys; scale by 1/√4 = 0.5.

Annotate

Walk the callouts on Worked example: biased attention for node 0 (C). Each one is a place this is easy to get subtly wrong.

• Each node i is a 4-dim one-hot (atom type), linearly projected to d_model=8 then split into d_k=4 head queries/keys; scale by 1/√4 = 0.5.
• SPD[0,:] = [0,1,1,2] → phi = [0.00, -0.50, -0.50, -1.20]. Bias penalises distant nodes, pushing attention mass toward closer neighbours.
• softmax([0.065, -0.423, -0.594, -1.244]) = [0.4165, 0.2556, 0.2154, 0.1125]. Node 0 (self) dominates; distant H(3) receives least weight.

Discussion prompt

Full bias: b[i,j] = phi(SPD[i,j]) + b_edge[i,j]. For the toy edge (C→N), a single-bond one-hot [1,0,0] projects to d_k=4 via Wc: [0.1719, -0.3128, -0.3979, -0.2849] (verified).

That is stated as though it always holds. Do one of two things: produce a case where it fails, or say precisely what rules such a case out. "It just does" is not on the menu.

Hint: Hunt at the extremes first — zero, one, negative, empty, equal. If every extreme survives, the reason they survive is the proof.

Predict first

A student writes this, and it looks reasonable:

Wrong approach: only use phi(SPD[i,j]) as the bias. Skip edge encoding because the distance already tells you how far apart nodes are.

It is wrong. Say what breaks — and say it before you turn the page.

Correct: Two pairs with SPD=1 get identical bias regardless of bond type — C=O (double bond) is indistinguishable from C–O (single bond).

Correct approach: combine spatial encoding with edge encoding for the full structural bias.

Why: Two pairs with SPD=1 get identical bias regardless of bond type — C=O (double bond) is indistinguishable from C–O (single bond).

Discussion prompt

The rule this trap just fixed:

Edge encoding encodes the type of bonds along the path; SPD encodes distance. Together they give the attention head complete structural context — critical for molecular property prediction.

Now break it on purpose. Build a case that violates it and follow the consequences until something visibly fails. Where does the failure first show up — and would you have noticed it if you had not been looking?

Hint: The dangerous rules are the ones whose violation still produces an answer. If yours fails loudly, try to find one that fails quietly.

Comparison matrix

From Sparse variant: mask non-edge pairs: refill the Node 0 sees column from what you know. The rest of the table is as it appeared.

Predict first

The table runs: 0(C) — masked | 0.0000 | 0.5426 | 0.4574 | 0.0000 · 1(N) — masked | 0.5072 | 0.0000 | 0.0000 | 0.4928 · 2(O) — masked | 1.0000 | 0.0000 | 0.0000 | 0.0000

In Worked example: masked attention (hard mask), given the rows so far: what is the next one — the row where Node is 3(H) — masked?

Correct: 3(H) — masked | 0.0000 | 1.0000 | 0.0000 | 0.0000

Why: The relationship between the columns, not the individual numbers, is what generates the next row. Adding -1e9 before softmax makes exp(-1e9) ≈ 0, effectively zeroing non-neighbour attention after softmax.

Annotate

In Worked example: masked attention (hard mask), before reading the notes: mark where the time actually goes. Which line dominates?

• Adding -1e9 before softmax makes exp(-1e9) ≈ 0, effectively zeroing non-neighbour attention after softmax.
• After masking, O has one non-zero attention weight = 1.0 on C. Soft Graphormer gives O: 0.31 on C, 0.15 on N, 0.48 on O(self), 0.05 on H — the self-loop dominates in soft mode.

Predict first

A student writes this, and it looks reasonable:

Wrong: hard masking still lets distant nodes influence each other — information just propagates through intermediate nodes over multiple layers.

It is wrong. Say what breaks — and say it before you turn the page.

Correct: True in principle, but each layer only attends to 1-hop neighbours, so information from L hops away takes L layers — same bottleneck as GNNs, not global.

Correct: hard-mask Graph Transformer is equivalent to a GNN in its information-propagation radius per layer.

Why: True in principle, but each layer only attends to 1-hop neighbours, so information from L hops away takes L layers — same bottleneck as GNNs, not global.

Sort into buckets

Some of these hold up and some are the exact mistakes this lesson is built to prevent. Sort them.

Holds up

Fix a toy molecule: C(0)–N(1)–H(3) and C(0)–O(2). Adjacency gives degrees and a shortest-path distance (SPD) matrix that will drive all structural encodings.; Unlike positional encoding's absolute index, φ encodes graph distance — it is the same value regardless of node ordering in the sequence.; After the encoder, VNode's hidden state is read out for graph-level tasks (e.g. predicting molecular energy). Node-level tasks use the individual node representations.

Breaks

Wrong approach: only use phi(SPD[i,j]) as the bias. Skip edge encoding because the distance already tells you how far apart nodes are.; Wrong: hard masking still lets distant nodes influence each other — information just propagates through intermediate nodes over multiple layers.

sound

These are stated as this lesson states them — each one survives the edge cases Lesson 96: Graph Transformer & Graphormer puts it through.

flawed

Each of these is lifted from a trap in this deck: reasonable-sounding, and wrong in a way that only shows up once you rely on it.

Comparison matrix

From VNode: the global graph token: every row here is a choice with a cost. Fill the SPD to VNode column, then say which row you would actually pick and what you give up for it.

Sort into buckets

Sort these by Complexity/layer, from memory, without looking back at O(N²) attention vs sparse GNN. Telling them apart on the spot is the skill; the table is only where the answer happens to be written down.

O(|E|·d)

GCN/GIN; Graph Transformer (mask)

O(|E|·d·H)

GAT

O(N²·d)

Graphormer (soft bias)

g1

Complexity/layer is "O(|E|·d)" for GCN/GIN, Graph Transformer (mask) — that is what the table on "O(N²) attention vs sparse GNN" records, and it is the single property separating this group from the rest.

g2

Complexity/layer is "O(|E|·d·H)" for GAT — that is what the table on "O(N²) attention vs sparse GNN" records, and it is the single property separating this group from the rest.

g3

Complexity/layer is "O(N²·d)" for Graphormer (soft bias) — that is what the table on "O(N²) attention vs sparse GNN" records, and it is the single property separating this group from the rest.

Discussion prompt

Key signal: if your task requires correlating features of nodes that are multiple hops apart and your graph is small enough for O(N²) attention, a Graph Transformer is worth the cost.

That is stated as though it always holds. Do one of two things: produce a case where it fails, or say precisely what rules such a case out. "It just does" is not on the menu.

Hint: Hunt at the extremes first — zero, one, negative, empty, equal. If every extreme survives, the reason they survive is the proof.

Discussion prompt

Run Pattern: building a Graphormer attention layer with this step confiscated:

Edge encoding: for each pair (i,j) embed edges on shortest path; mean-pool → b_edge[i,j]

Is it still possible? If it is, say what takes its place and what it costs you. If it is not, say exactly what that step was providing that nothing else does.

Hint: A step you can drop for free was never load-bearing. If you cannot drop it, name the thing that goes wrong the moment it is gone.

Discussion prompt

Pattern: building a Graphormer attention layer works on the cases you have just seen. Push it to the edge: what is the most degenerate input it still handles — empty, zero, one item, everything equal — and what is the first case where it stops being true? Name the case, not just "it breaks".

Hint: Try the smallest legal input, then the largest, then the one where two things collide. Methods are specified at their edges; the middle takes care of itself.

Eliminate the wrong options

In the toy graph, node O(2) to H(3) has SPD=3 and phi=-2.10; O to C(0) has SPD=1 and phi=-0.50. Assuming equal raw attention scores, which statement is correct?

3 of these 4 are wrong. Strike them one at a time, and say what rules each one out before you strike the next. The survivor is the answer.

• A. O attends to H with higher weight than to C, because a larger |phi| value means stronger attention.
• B. O attends to C with higher weight, because phi=-0.50 is a smaller penalty than phi=-2.10, leaving a higher biased logit.
• C. O attends to H and C equally, because phi values cancel out after softmax normalization.
• D. O only attends to its 1-hop neighbour C; the mask makes H receive zero weight.

Survives elimination: B

Why: phi(1)=-0.50 is a weaker penalty than phi(3)=-2.10. With equal raw scores, the biased logit for (O→C) is 1.70 higher than for (O→H), so softmax assigns more weight to C. phi encodes graph proximity, not strength of bonding.

Check your understanding

In the toy graph, node O(2) to H(3) has SPD=3 and phi=-2.10; O to C(0) has SPD=1 and phi=-0.50. Assuming equal raw attention scores, which statement is correct?

• A. O attends to H with higher weight than to C, because a larger |phi| value means stronger attention.
• B. O attends to C with higher weight, because phi=-0.50 is a smaller penalty than phi=-2.10, leaving a higher biased logit. (correct)
• C. O attends to H and C equally, because phi values cancel out after softmax normalization.
• D. O only attends to its 1-hop neighbour C; the mask makes H receive zero weight.

Answer: B

Why: phi(1)=-0.50 is a weaker penalty than phi(3)=-2.10. With equal raw scores, the biased logit for (O→C) is 1.70 higher than for (O→H), so softmax assigns more weight to C. phi encodes graph proximity, not strength of bonding.

Why A tempts people

Larger |phi| means a more negative additive bias, which lowers the logit and therefore lowers attention weight — not raises it.

Why C tempts people

Softmax is order-preserving on its inputs: different input logits produce different output weights. Only identical inputs cancel.

Why D tempts people

D describes the hard-mask variant (Graph Transformer), not Graphormer's soft-bias approach. In Graphormer, non-adjacent pairs get a bias penalty, not -∞.

Predict first

Which of the following correctly describes a design choice in Graphormer?

Answer it in your own words, now, with nothing to choose from. The options are on the next slide — and picking the right one off a list is an easier skill than producing it.

Correct: The VNode is connected to all real nodes with a fixed SPD=1, so phi(1) is added to every VNode–node attention logit.

Why: Graphormer assigns SPD(VNode, i)=1 for all real nodes i. This means phi(1) biases the VNode's attention toward all nodes uniformly, giving it a balanced global view. The VNode hidden state after the encoder is used as the graph-level representation.

Check your understanding

Which of the following correctly describes a design choice in Graphormer?

• A. The VNode is connected to all real nodes with SPD=0 so it acts as a self-loop.
• B. Centrality encoding uses eigenvector centrality because it captures global importance better than degree.
• C. The VNode is connected to all real nodes with a fixed SPD=1, so phi(1) is added to every VNode–node attention logit. (correct)
• D. Centrality encoding is added to the attention logits, not to the node feature vectors.

Answer: C

Why: Graphormer assigns SPD(VNode, i)=1 for all real nodes i. This means phi(1) biases the VNode's attention toward all nodes uniformly, giving it a balanced global view. The VNode hidden state after the encoder is used as the graph-level representation.

Why A tempts people

SPD=0 would mean the VNode treats itself as the same node — that conflicts with the shortest-path semantics and would make phi(0)=0 remove all bias for VNode interactions.

Why B tempts people

Graphormer uses in-degree and out-degree (simple degree centrality), not eigenvector centrality — degree is cheaper to compute and worked well in practice on molecular benchmarks.

Why D tempts people

Centrality encoding is added to the input node representation h_i⁰, not to attention logits. Spatial encoding is what modifies attention logits.

Eliminate the wrong options

For N=50, |E|=120, d=128, how many times more expensive (approximately) is one Graphormer layer vs one GCN layer in terms of multiply-accumulate operations?

3 of these 4 are wrong. Strike them one at a time, and say what rules each one out before you strike the next. The survivor is the answer.

• A. About 2× — both are roughly linear in N.
• B. About 26× — Graphormer O(N²·d)=320,000 vs GCN O(|E|·d)=15,360.
• C. About 50× — because Graphormer needs N² pairs while GCN needs |E|=2N pairs.
• D. Identical — Graphormer masks to edges anyway, making it O(|E|·d).

Survives elimination: B

Why: Graphormer: O(N²·d) = 50²×128 = 320,000. GCN: O(|E|·d) = 120×128 = 15,360. Ratio = 320000/15360 ≈ 20.8×, closest to option B's ~26× (the approximation difference comes from ignoring constant factors). The key point: Graphormer's cost scales quadratically in N, GCN's scales with edge count.

Check your understanding

For N=50, |E|=120, d=128, how many times more expensive (approximately) is one Graphormer layer vs one GCN layer in terms of multiply-accumulate operations?

• A. About 2× — both are roughly linear in N.
• B. About 26× — Graphormer O(N²·d)=320,000 vs GCN O(|E|·d)=15,360. (correct)
• C. About 50× — because Graphormer needs N² pairs while GCN needs |E|=2N pairs.
• D. Identical — Graphormer masks to edges anyway, making it O(|E|·d).

Answer: B

Why: Graphormer: O(N²·d) = 50²×128 = 320,000. GCN: O(|E|·d) = 120×128 = 15,360. Ratio = 320000/15360 ≈ 20.8×, closest to option B's ~26× (the approximation difference comes from ignoring constant factors). The key point: Graphormer's cost scales quadratically in N, GCN's scales with edge count.

Why A tempts people

Graphormer's all-pairs attention is O(N²), not O(N). Even for modest N=50, the quadratic term dominates.

Why C tempts people

50× would be O(N²)/O(N·d)·constant, but |E|=120 ≠ 2N=100 in general. The exact ratio depends on graph sparsity; 26× is the computed value for this graph.

Why D tempts people

D describes the hard-mask sparse Graph Transformer, not Graphormer. Graphormer uses a soft additive bias (never -∞), so it computes all N² attention pairs.

Predict first

The table runs: C (0) | val[0] | val[1] | val[2] | val[3] · Shape | (4, 4) | — | — | —

In Full Graphormer mini-module in PyTorch, given the rows so far: what is the next one — the row where Node is Module params?

Correct: Module params | 3×(8×4)+5×1 | = | 101 | params

Why: The relationship between the columns, not the individual numbers, is what generates the next row. The SPD matrix is a precomputed graph property (BFS); we convert it to bias via a learned embedding table phi_table of size max_dist+1.

Comparison matrix

From Full Graphormer mini-module in PyTorch: refill the out[2] column from what you know. The rest of the table is as it appeared.

Put in order

Put the moves of Your turn — Graphormer attention from scratch into the order they have to happen.

1. Milestone 1 — compute SPD for a path graph of 5 nodes
2. Milestone 2 — build the phi table and spatial bias matrix B
3. Milestone 3 — run biased attention for node 0; print attn_weights[0] and verify attn[0,1] > attn[0,4]
4. Show it off: print the full 5×5 attention weight matrix and verify the diagonal dominance pattern along rows

Why: These are the moves of the worked example in the order it makes them, and each one is set up by the one before it. For a path 0–1–2–3–4, SPD[i,j] = |i−j|.

Annotate

Walk the callouts on Your turn — Graphormer attention from scratch. Each one is a place this is easy to get subtly wrong.

• For a path 0–1–2–3–4, SPD[i,j] = |i−j|. Write the 5×5 SPD matrix and confirm the diagonal is 0 and the corners SPD[0,4]=SPD[4,0]=4.
• Use phi_weights = [0.0, -0.5, -1.2, -2.1, -3.0] (same as lesson). Construct a 5×5 tensor B where B[i,j]=phi(min(|i-j|,4)). Check B[0,4]=B[4,0]=-3.0.
• Node 0 has SPD=1 to node 1 and SPD=4 to node 4. phi(1)=-0.50 vs phi(4)=-3.00: a 2.50 penalty differential guarantees more weight on node 1 (barring extreme raw score differences).

Draw it

One page, no notation unless you need it: draw how these connect — Why graphs? Transformers already are graphs · Graphormer's three structural encodings · Soft bias vs hard mask — two Graph Transformer flavours · Complexity, scaling, applications. Put an arrow wherever one of them is what makes another possible, and label the arrow with why.