Why Is My GPU Waiting for the Next Batch?

Lesson

Start with eight pinned CPU batches and a CUDA projection. Run the serial baseline, then prefetch the next batch on a copy stream while the current batch computes. Keep an event per copied tensor, wait on that event before consuming it, and record its use on the compute stream so the allocator cannot recycle storage early. Pinning and input creation are outside timing; the timed work includes H2D copies, projection and output assembly. non_blocking=True alone does not prove overlap. Compare the Trace copy/compute intervals, H2D bytes/count, synchronized batch-loop latency and samples/sec. Hardware copy engines and workload size determine whether overlap helps.

Measurement contract: the real read-only driver defines WORKLOAD. Two warmups precede five paired trials with alternating baseline/candidate order. Each trial uses fresh deterministic inputs. Synchronized host time includes solve and completion, excludes input construction and reference verification. METRICS reports every sample, median and IQR; memory includes persistent buffers and allocator state. One additional solve is isolated in judged for Trace. No speedup is promised. Run the unchanged starter first.

What you will learn

• Identify the bottleneck from the appropriate evidence.
• Preserve the output contract on fresh inputs.
• Compare paired measurements and numerical error.

Starter code

    return baseline(case)