Provider rate limits are enforced as token buckets on requests per minute (RPM) and
tokens per minute (TPM). Push too many parallel workers and you get a wall of
429 retries; too few and you leave throughput on the table. Enter your request profile
and account limits below to size the maximum safe concurrency, see which limit is the bottleneck,
and estimate the 429/queue-wait risk if you over-provision.
Every request costs one unit against the RPM bucket and
input + output tokens against the TPM bucket. To stay under a per-minute limit,
your sustained throughput can be no higher than the limit divided by the per-request cost. So the two
independent throughput ceilings, after applying your safety factor s, are:
tokensPerReq = inputTok*(1 - cache) + inputTok*cache*0.25 + outputTok rpsFromRPM = (RPM * s) / 60 rpsFromTPM = (TPM * s) / tokensPerReq / 60 sustainRPS = min(rpsFromRPM, rpsFromTPM) // the bottleneck
The second half is Little's Law: for a stable system the average number of
requests in flight equals arrival rate times the time each spends in the system
(L = λ × W). Here W is your mean latency, and the maximum
arrival rate you can safely feed is sustainRPS, so the safe concurrency is
maxWorkers = floor(sustainRPS × latency). Adding workers beyond that number does
not increase completed throughput — it only grows the queue, and once the token bucket empties the
provider returns 429. Cached input tokens are weighted at roughly 0.25× because
most providers bill and rate-limit cache reads at a discount. The 429/queue-wait risk
is estimated from the overshoot ratio planned / maxWorkers: each excess worker's requests
must wait for bucket refill, so expected added wait ≈ (overshoot - 1) × latency,
and risk rises steeply as you exceed the ceiling. Sizing to 80–90% utilization absorbs latency
variance and retry bursts without tripping the bucket.