My Model Recompiles When the Batch Size Changes

Lesson

The request batches are [7,16,9,32,7,24], totaling 95 rows. The starter forces specialization with dynamic=False. Try dynamic=True or pad to a bucket of 32 and slice outputs back to the true row count. The model reduces only within each row, so padding other rows cannot affect valid outputs. Keep the request distribution identical. The counting backend reports actual compiler callback requests, while total cold latency includes compilation. Per-request synchronized p50/p95 and padded rows expose the cost. Warmed paired timings compare against eager execution; dynamic shapes are not universally faster. Compiler diagnostics are CPU evidence, not a conclusion drawn from GPU timeline gaps.

Measurement contract: the real read-only driver defines WORKLOAD. Two warmups precede five paired trials with alternating baseline/candidate order. Each trial uses fresh deterministic inputs. Synchronized host time includes solve and completion, excludes input construction and reference verification. METRICS reports every sample, median and IQR; memory includes persistent buffers and allocator state. One additional solve is isolated in judged for Trace. No speedup is promised. Run the unchanged starter first.

What you will learn

• Identify the bottleneck from the appropriate evidence.
• Preserve the output contract on fresh inputs.
• Compare paired measurements and numerical error.

Starter code

    state = case['_state']
    if 'compiled' not in state:
        state['compiled'] = torch.compile(model, dynamic=False, backend=compiler(case))
    return run_requests(case, state['compiled'])