Break a streamed LLM response into its two real cost centers — time-to-first-token (TTFT) and per-token generation time — then project the p50, p90, p95 and p99 total latency and check every percentile against your service-level objective. Nothing is uploaded; all math runs in your browser.
| Pctl | Total ms | vs SLO | Margin |
|---|
A streamed LLM response has two additive components. First is time-to-first-token (TTFT): queueing, prompt tokenization and the prefill forward pass before any output appears. Second is the generation phase, where tokens are decoded one at a time. The base end-to-end latency is simply:
Median numbers alone hide the problem SLOs exist to catch: the tail. Under concurrency, batching contention and cold caches, the slowest requests balloon. Rather than assume a fixed pad, this tool treats total latency as roughly log-normal — the standard shape for positive, right-skewed latency. From your median and a tail multiplier m = p99 / p50, it recovers the distribution's shape parameter σ = ln(m) / z0.99, where z0.99 ≈ 2.326 is the 99th-percentile standard normal quantile. Any percentile then follows:
with zp the normal quantile for that percentile (z₀.₉₀≈1.282, z₀.₉₅≈1.645). When the SLO is measured at first token, only the TTFT component is scaled; when it governs the full response, the whole total is scaled, since prefill and decode tails correlate under shared load. The tool then compares the latency at your chosen percentile to the budget and reports the margin — negative means you are over budget. Finally it inverts the equation to solve for the largest output_tokens whose target-percentile latency still lands under the SLO, so you can cap max_tokens defensively instead of discovering breaches in production. Everything recomputes live as you type.