Throughput

Throughput is how much work a system completes per unit of time, such as tokens generated per second across all users. Batching many requests together raises throughput but can add latency for each one.

Operators balance throughput, which lowers cost per request, against latency, which affects user experience.