Faster Serving, Slower Users?

Lesson

Choose a maximum batch size and maximum wait for a deterministic sixteen-request arrival trace. serve implements arrival waits, queueing and four sequential synthetic token projections per request. The output must stay in request order. Compare tokens/sec, time to first token, inter-token time, p95 request latency and peak memory. The illustrative 100 ms SLO is reported, not a calibrated gate. This small local synthetic server omits network and real autoregressive dependencies. Throughput improvement alone does not prove latency improvement; change max_batch and max_wait_ms and compare the same arrivals.

Measurement contract: the real read-only driver defines WORKLOAD. Two warmups precede five paired trials with alternating baseline/candidate order. Each trial uses fresh deterministic inputs. Synchronized host time includes solve and completion, excludes input construction and reference verification. METRICS reports every sample, median and IQR; memory includes persistent buffers and allocator state. One additional solve is isolated in judged for Trace. No speedup is promised. Run the unchanged starter first.

What you will learn

• Identify the bottleneck from the appropriate evidence.
• Preserve the output contract on fresh inputs.
• Compare paired measurements and numerical error.

Starter code

    return serve(case, max_batch=1, max_wait_ms=0)