Same Kernels, Fewer Python Submissions: CUDA Graphs

Lesson

Capture sixteen static-shape inference steps after warmup on a side stream. Hold the graph and its input/output tensors in case['_state']; these addresses must remain alive. Each invocation receives new input values: copy them into the static input before replay. Return a clone if later replay would overwrite a result that the caller retains. Cold latency includes setup/capture; warmed timings include the input update, replay and returned output. Compare host submission gaps, latency and additional allocation. Capture contains no scalar reads or host synchronization. Graph replay reduces host submission overhead; operator fusion is a different mechanism.

Measurement contract: the real read-only driver defines WORKLOAD. Two warmups precede five paired trials with alternating baseline/candidate order. Each trial uses fresh deterministic inputs. Synchronized host time includes solve and completion, excludes input construction and reference verification. METRICS reports every sample, median and IQR; memory includes persistent buffers and allocator state. One additional solve is isolated in judged for Trace. No speedup is promised. Run the unchanged starter first.

What you will learn

• Identify the bottleneck from the appropriate evidence.
• Preserve the output contract on fresh inputs.
• Compare paired measurements and numerical error.

Starter code

    return baseline(case)