Decoder Block 2: The CUDA Pipeline

Lesson

Stage 1 proved you know what a decoder block computes. This stage makes the GPU compute it, as a deliberately clear pipeline: one kernel per operator, one launch per step, every intermediate visible in memory and on the profiler timeline. This is how llama.cpp and every serious inference engine BEGAN, before any fusion: a correct multi-kernel baseline you can trust while you optimize.

The editor now holds three files. main.cu is the trusted driver and is read-only: it owns the packed weights, the RoPE tables, the workspace map, the fixed launch order, and the same nine-checkpoint judgement as stage 1, poison included. kernels.cuh fixes the model constants and the six kernel signatures. The starter provides six correct bodies in kernels.cu. Profile it first, then optimize one operator at a time:

1. rmsnormKernel, one block per token row: the shared-memory tree you built in the Reduction path, ending in a scaled write.
2. matmulKernel, one thread per output element. The driver launches this SEVEN times: Q, K, V, the output projection, gate, up, and down. Same code, different pointers and dimensions. Watch for it on the timeline.
3. ropeKernel, one thread per (token, head, frequency): rotate the half-split pair (f, f + 32). The driver runs it for Q's eight heads and K's two; V never sees it.
4. attentionKernel, one thread per (query row, query head): causal, grouped (kv_head = q_head / 4), max-subtracted softmax. This baseline recomputes dot products in three passes so it needs no score storage; it is honest and slow, and stages 5 and 6 will make you feel exactly how slow.
5. residualAddKernel and swigluKernel, plain elementwise operators.

Fourteen launches per verified sequence length, T = 1, 4, 17, 33 and 256. The odd lengths are there on purpose: nothing in your indexing may assume a convenient multiple. After Run, open the timeline and count the launches; that picture, one bar per operator with gaps between them, is the thing every later stage of this path will shrink.

Optimize a correct baseline. The original starter now implements all six operators. Run it unchanged, then change one operator at a time and inspect the first failing checkpoint. Saved drafts are not overwritten.

What you will learn

• Decompose the decoder block into per-operator kernels with explicit intermediate tensors.
• Reuse one generic GEMM kernel across seven differently-shaped projection launches.
• Keep RoPE rotation and the GQA head mapping as index arithmetic on the GPU.
• Write a causal, numerically stable attention kernel that needs no score storage.
• Read a multi-kernel pipeline's timeline: launch count, per-operator cost, and the gaps between.