Inference server: batch the requests
A decoder serves many requests, but the naive step advances the sequences one at a time, so each tiny kernel uses a sliver of the GPU and the per-step latency is paid once per sequence. Profile the run, then advance the whole batch in one launch to fill the GPU and raise throughput without changing the output.
This project is available to study. Hosted execution is not currently available.
Languages: CUDA C++
Lesson
The trace shows many tiny kernels advancing independent sequences one at a time. The server holds many independent sequences, but the naive step runs a per-sequence matmul+bias+ReLU followed by a normalize, looped over the batch. Each kernel uses one small block—a fraction of one SM—so the GPU is almost entirely idle. Because each token depends on the previous one, the step is latency-bound rather than compute-bound. Look for work that is serialized even though the requests do not depend on one another.
This synthetic decoder reports aggregate throughput. It does not include network/queueing latency and does not establish a per-request SLO. Continue with the Python serving-latency lab for that distinction.
What you will learn
- Recognize an underutilization bottleneck: at batch=1 each step is a tiny single-block kernel and the GPU sits nearly idle, latency-bound on the autoregressive dependency.
- Advance all sequences in one launch (batch*dim threads) so independent sequences run in parallel and the per-step cost is amortized across the batch.
- Measure how batching turns a latency-bound decoder into a throughput-bound one (tokens/sec scaling with batch size).