Lesson 94: Graph Neural Networks (GCN & GAT)

Discussion prompt

Before we open Lesson 94: Graph Neural Networks (GCN & GAT): without looking back, what was the main idea of BERT Variants — XLNet, RoBERTa, T5, DeBERTa, ALBERT, and what could you do by the end of it that you could not do before?

Hint: One sentence for the idea, one for the skill. If the second one is blank, that is the part to revisit.

Sort into buckets

Sort these by typical task, from memory, without looking back at Graphs: nodes, edges, features. Telling them apart on the spot is the skill; the table is only where the answer happens to be written down.

node classification

social network; citation network

graph classification

molecular graph

link prediction

knowledge graph

g1

typical task is "node classification" for social network, citation network — that is what the table on "Graphs: nodes, edges, features" records, and it is the single property separating this group from the rest.

g2

typical task is "graph classification" for molecular graph — that is what the table on "Graphs: nodes, edges, features" records, and it is the single property separating this group from the rest.

g3

typical task is "link prediction" for knowledge graph — that is what the table on "Graphs: nodes, edges, features" records, and it is the single property separating this group from the rest.

Comparison matrix

From The message-passing framework: refill the AGGREGATE column from what you know. The rest of the table is as it appeared.

Predict first

Graph: nodes 0-3, edges {(0,1),(1,2),(2,3),(0,3)} — a 4-cycle. Add self-loops to get Ã. Compute degree matrix D̃ and then  = D̃^{-1/2} à D̃^{-1/2}. Predict each diagonal entry before proceeding.

Commit before you compute: what does Compute  on a 4-cycle come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: Â[i,i] = 1/sqrt(3·3) = 1/3 ≈ 0.3333; Â[i,j] (connected) = 1/sqrt(3·3) ≈ 0.3333 too

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. Because all degrees are equal (regular graph), all non-zero entries of  are the same: 1/(d̃_u^{1/2} · d̃_v^{1/2}) = 1/3.

Comparison matrix

From Compute  on a 4-cycle: every row here is a choice with a cost. Fill the value column, then say which row you would actually pick and what you give up for it.

Predict first

One GCN layer: H^(1) = ReLU(Â H^(0) W). Use the  from the 4-cycle, random H^(0) (4×2), and random W (2×3). Trace node 0's output vector.

Commit before you compute: what does GCN forward pass: Â H W then ReLU come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: A_norm @ H: each row i of the output is (1/3)(h_i + sum of neighbor features)

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. For node 0, neighbors are {1, 3} plus self, so row 0 of A_norm@H = (h_0 + h_1 + h_3)/3.

Discussion prompt

Work backwards. The example finished here:

A_norm @ H: each row i of the output is (1/3)(h_i + sum of neighbor features)

What was it asked to do, and what must it have been given? Reconstruct the problem from its answer.

Hint: Every quantity in the result had to enter somewhere. Account for each one.

Predict first

A student writes this, and it looks reasonable:

Use the raw adjacency A (without self-loop) for normalization:  = D^{-1/2} A D^{-1/2}.

It is wrong. Say what breaks — and say it before you turn the page.

Correct: Without self-loops, node v's own features h_v are NOT included in the aggregation — the new embedding h_v^{(l+1)} depends only on neighbors, not on v itself.

Always add self-loops first: à = A + I. Compute degree on Ã, then normalize:  = D̃^{-1/2} à D̃^{-1/2}.

Why: Without self-loops, node v's own features h_v are NOT included in the aggregation — the new embedding h_v^{(l+1)} depends only on neighbors, not on v itself. The node forgets where it started.

Discussion prompt

The rule this trap just fixed:

With self-loops, row i of ·H includes h_i itself at weight 1/d̃_i. This ensures the node's own representation propagates forward — critical for learning node-level properties.

Now break it on purpose. Build a case that violates it and follow the consequences until something visibly fails. Where does the failure first show up — and would you have noticed it if you had not been looking?

Hint: The dangerous rules are the ones whose violation still produces an answer. If yours fails loudly, try to find one that fails quietly.

Discussion prompt

GCN treats all neighbors equally: weight = 1/sqrt(d̃_u · d̃_v). GAT (Velickovic 2018) lets the model learn how much attention to pay each neighbor, conditioned on the features of both endpoints.

That is stated as though it always holds. Do one of two things: produce a case where it fails, or say precisely what rules such a case out. "It just does" is not on the menu.

Hint: Hunt at the extremes first — zero, one, negative, empty, equal. If every extreme survives, the reason they survive is the proof.

Predict first

Node 0 has neighbors {0 (self), 1, 3} in the 4-cycle. Compute raw attention scores e_{0j} for each neighbor, then apply softmax. Verify the weights sum to 1.

Commit before you compute: what does GAT attention weights on a toy neighborhood come out to? A rough magnitude and the right form is enough — the point is to have something concrete to be wrong about.

Correct: Raw scores: e_{0,0}=1.4656, e_{0,1}=3.0661, e_{0,3}=1.0771

Why: A prediction you can defend turns the computation into a check rather than a leap of faith — and an answer that contradicts it is caught on the spot. Each e_{0,j} = LeakyReLU(a^T [Wh_0 || Wh_j]).

Discussion prompt

Work backwards. The example finished here:

Raw scores: e_{0,0}=1.4656, e_{0,1}=3.0661, e_{0,3}=1.0771

What was it asked to do, and what must it have been given? Reconstruct the problem from its answer.

Hint: Every quantity in the result had to enter somewhere. Account for each one.

Predict first

A student writes this, and it looks reasonable:

Apply softmax over all N nodes to get α_{uv}: alpha = softmax(e[v, :]) where e has shape (N,).

It is wrong. Say what breaks — and say it before you turn the page.

Correct: Each node has only a few neighbors, but softmax over all N nodes will assign tiny but non-zero weights to non-connected nodes, leaking information across non-edges and violating the local aggregation principle of GNNs.

Mask non-neighbors with -inf before softmax so they receive exactly zero attention weight.

Why: Each node has only a few neighbors, but softmax over all N nodes will assign tiny but non-zero weights to non-connected nodes, leaking information across non-edges and violating the local aggregation principle of GNNs.

Sort into buckets

Some of these hold up and some are the exact mistakes this lesson is built to prevent. Sort them.

Holds up

All major GNNs (GCN, GAT, GraphSAGE, GIN) are instances of the same message-passing template — only the choices of MESSAGE, AGGREGATE, and UPDATE differ.; Node classification uses H^(L)[v] directly. For graph classification, you need one fixed-size vector per graph — a readout (global pooling) over all node embeddings.; Build TinyGCN — two GCN layers (no bias, 4→8→2) on an 8-node synthetic graph with two 4-cliques connected by a bridge. Task: classify each node into community 0 or 1.

Breaks

Use the raw adjacency A (without self-loop) for normalization:  = D^{-1/2} A D^{-1/2}.; Apply softmax over all N nodes to get α_{uv}: alpha = softmax(e[v, :]) where e has shape (N,).

sound

These are stated as this lesson states them — each one survives the edge cases Lesson 94: Graph Neural Networks (GCN & GAT) puts it through.

flawed

Each of these is lifted from a trap in this deck: reasonable-sounding, and wrong in a way that only shows up once you rely on it.

Comparison matrix

From From node embeddings to a graph vector: refill the differentiable? column from what you know. The rest of the table is as it appeared.

Discussion prompt

Run The GCN/GAT recipe with this step confiscated:

Depth: 2-3 layers typical; L layers → L-hop neighborhood receptive field

Is it still possible? If it is, say what takes its place and what it costs you. If it is not, say exactly what that step was providing that nothing else does.

Hint: A step you can drop for free was never load-bearing. If you cannot drop it, name the thing that goes wrong the moment it is gone.

Discussion prompt

The GCN/GAT recipe works on the cases you have just seen. Push it to the edge: what is the most degenerate input it still handles — empty, zero, one item, everything equal — and what is the first case where it stops being true? Name the case, not just "it breaks".

Hint: Try the smallest legal input, then the largest, then the one where two things collide. Methods are specified at their edges; the middle takes care of itself.

Eliminate the wrong options

In a graph where node u has degree 4 (after adding self-loop, d̃_u = 5) and node v has degree 2 (d̃_v = 3), and (u,v) is an edge, what is Â[u,v] in the GCN normalized adjacency?

3 of these 4 are wrong. Strike them one at a time, and say what rules each one out before you strike the next. The survivor is the answer.

• A. 1 / sqrt(5 × 3) ≈ 0.2582
• B. 1 / (5 + 3) = 0.125
• C. 2 / (5 + 3) = 0.25
• D. 1 / sqrt(4 × 2) ≈ 0.3536

Survives elimination: A

Why: Â[u,v] = 1/(d̃_u^{1/2} · d̃_v^{1/2}) = 1/sqrt(5×3) = 1/sqrt(15) ≈ 0.2582. The formula uses the degree after adding self-loops (d̃), not the original degree (d).

Check your understanding

In a graph where node u has degree 4 (after adding self-loop, d̃_u = 5) and node v has degree 2 (d̃_v = 3), and (u,v) is an edge, what is Â[u,v] in the GCN normalized adjacency?

• A. 1 / sqrt(5 × 3) ≈ 0.2582 (correct)
• B. 1 / (5 + 3) = 0.125
• C. 2 / (5 + 3) = 0.25
• D. 1 / sqrt(4 × 2) ≈ 0.3536

Answer: A

Why: Â[u,v] = 1/(d̃_u^{1/2} · d̃_v^{1/2}) = 1/sqrt(5×3) = 1/sqrt(15) ≈ 0.2582. The formula uses the degree after adding self-loops (d̃), not the original degree (d).

Why B tempts people

1/(d̃_u + d̃_v) is the arithmetic-mean normalization, not the symmetric geometric-mean used by GCN. GCN uses the product under a square root.

Why C tempts people

2/(d̃_u + d̃_v) has no derivation from the Kipf & Welling formula; the factor of 2 appears nowhere in D̃^{-1/2} Ã D̃^{-1/2}.

Why D tempts people

Using the original degrees (4 and 2) instead of the self-loop-augmented degrees (5 and 3) is the 'forgot self-loop in denominator' mistake — the trap slide covers exactly this error.

Predict first

Node v has 3 neighbors (plus itself, so 4 total in N(v)∪{v}). After computing GAT attention weights α_{uv} for all u ∈ N(v)∪{v}, what must be true?

Answer it in your own words, now, with nothing to choose from. The options are on the next slide — and picking the right one off a list is an easier skill than producing it.

Correct: Σ_{u ∈ N(v)∪{v}} α_{uv} = 1

Why: The softmax in GAT is applied only over the neighborhood N(v)∪{v}, making the 4 weights a proper probability distribution that sums to 1. Non-neighbors receive exactly 0 weight (via -inf masking). This was verified in code: all row sums equal 1.0.

Check your understanding

Node v has 3 neighbors (plus itself, so 4 total in N(v)∪{v}). After computing GAT attention weights α_{uv} for all u ∈ N(v)∪{v}, what must be true?

• A. Σ_{u ∈ N(v)∪{v}} α_{uv} = 1 (correct)
• B. Each α_{uv} ∈ [0, 1] and Σ_{u ∈ V} α_{uv} = 1 (sum over all N nodes)
• C. Σ_{u ∈ N(v)∪{v}} α_{uv} = 1/4 (one contribution per neighbor)
• D. Σ_{u ∈ N(v)∪{v}} α_{uv} = K where K is the number of attention heads

Answer: A

Why: The softmax in GAT is applied only over the neighborhood N(v)∪{v}, making the 4 weights a proper probability distribution that sums to 1. Non-neighbors receive exactly 0 weight (via -inf masking). This was verified in code: all row sums equal 1.0.

Why B tempts people

Summing over all N nodes would mean non-neighbors could receive positive weight, violating the local aggregation principle. The correct softmax domain is N(v)∪{v}, not all of V.

Why C tempts people

α_{uv} = 1/4 for each neighbor would be uniform weighting — identical to mean pooling. GAT's entire purpose is to learn non-uniform weights; a uniform constraint defeats that.

Why D tempts people

Multi-head GAT runs K independent attention functions and concatenates their outputs, but each head's weights still sum to 1 independently. K appears in the output dimension, not in the weight sum.

Eliminate the wrong options

You are building a citation-network node classifier. Each paper has a bag-of-words feature vector. Some papers cite far more relevant works than others. Which statement correctly describes when GAT has an advantage over GCN?

3 of these 4 are wrong. Strike them one at a time, and say what rules each one out before you strike the next. The survivor is the answer.

• A. GAT can up-weight the most informative neighboring papers and ignore noisy citations, while GCN averages all neighbors equally weighted by degree.
• B. GAT is always better than GCN because attention adds parameters, and more parameters always improve accuracy.
• C. GCN is better when neighbor features vary widely; GAT is better when all neighbors are equally informative.
• D. GAT removes the need for the self-loop trick because attention automatically down-weights the node's own features.

Survives elimination: A

Why: GAT learns feature-conditioned attention weights α_{uv}, so it can assign high weight to topically relevant neighbors and low weight to off-topic citations — exactly what fixed degree-normalization cannot do. On Cora, GAT historically outperforms GCN by ~1–2% because the benefit of selective attention outweighs the added parameters.

Check your understanding

You are building a citation-network node classifier. Each paper has a bag-of-words feature vector. Some papers cite far more relevant works than others. Which statement correctly describes when GAT has an advantage over GCN?

• A. GAT can up-weight the most informative neighboring papers and ignore noisy citations, while GCN averages all neighbors equally weighted by degree. (correct)
• B. GAT is always better than GCN because attention adds parameters, and more parameters always improve accuracy.
• C. GCN is better when neighbor features vary widely; GAT is better when all neighbors are equally informative.
• D. GAT removes the need for the self-loop trick because attention automatically down-weights the node's own features.

Answer: A

Why: GAT learns feature-conditioned attention weights α_{uv}, so it can assign high weight to topically relevant neighbors and low weight to off-topic citations — exactly what fixed degree-normalization cannot do. On Cora, GAT historically outperforms GCN by ~1–2% because the benefit of selective attention outweighs the added parameters.

Why B tempts people

More parameters do not automatically help — they can overfit. GAT's advantage is the inductive bias of edge-wise attention, not raw parameter count. On very small graphs GAT can overfit where GCN does not.

Why C tempts people

This is backwards. GCN is best when neighbors are roughly equally informative (uniform weights are near-optimal). GAT shines precisely when neighbor quality varies — it can selectively up-weight informative neighbors.

Why D tempts people

Both GCN and GAT typically use self-loops. GAT's attention weight for the self-edge (u=v) is computed the same way as for other edges and is learned, but the self-loop is still explicitly included in N(v)∪{v}.

Discussion prompt

Build TinyGCN — two GCN layers (no bias, 4→8→2) on an 8-node synthetic graph with two 4-cliques connected by a bridge. Task: classify each node into community 0 or 1.

That is stated as though it always holds. Do one of two things: produce a case where it fails, or say precisely what rules such a case out. "It just does" is not on the menu.

Hint: Hunt at the extremes first — zero, one, negative, empty, equal. If every extreme survives, the reason they survive is the proof.

Sort into buckets

Sort these by output shape, from memory, without looking back at Milestone 2 — TinyGCN forward pass. Telling them apart on the spot is the skill; the table is only where the answer happens to be written down.

(8, 8)

W1 (Linear 4→8); ReLU + A_norm@

(8, 2)

W2 (Linear 8→2); A_norm@ (readout)

—

TOTAL

g1

output shape is "(8, 8)" for W1 (Linear 4→8), ReLU + A_norm@ — that is what the table on "Milestone 2 — TinyGCN forward pass" records, and it is the single property separating this group from the rest.

g2

output shape is "(8, 2)" for W2 (Linear 8→2), A_norm@ (readout) — that is what the table on "Milestone 2 — TinyGCN forward pass" records, and it is the single property separating this group from the rest.

g3

output shape is "—" for TOTAL — that is what the table on "Milestone 2 — TinyGCN forward pass" records, and it is the single property separating this group from the rest.

Comparison matrix

From Milestone 3 — train to 100% accuracy: every row here is a choice with a cost. Fill the accuracy column, then say which row you would actually pick and what you give up for it.

Comparison matrix

From The full program: refill the value column from what you know. The rest of the table is as it appeared.

Draw it

One page, no notation unless you need it: draw how these connect — Message passing — the GNN abstraction · GCN — normalized adjacency · GAT — learned edge attention · Readout — graph-level pooling · Your turn: TinyGCN from scratch. Put an arrow wherever one of them is what makes another possible, and label the arrow with why.