LLM Time to First Token and Throughput Explained

Measuring time to first token (TTFT) and throughput shows how quickly an LLM deployment begins responding and how much text it generates over time. You will send controlled requests, record timestamps and token counts, then compare results under a stated workload; this helps reveal latency, capacity limits and regressions.

What You’ll Need

• Access to the deployed model’s API or serving endpoint, including any required authentication.
• A client or test script that can capture request start time, first streamed token time, completion time and token counts.
• A tokenizer compatible with the served model, or reliable usage fields from the API for generated-token counts.
• A representative prompt set and a safe test environment. For load tests, coordinate with the service owner and avoid disrupting production users.
• A place to save results, such as a CSV file, monitoring dashboard or benchmark report.

Step-by-Step

1. Define the measurements. TTFT is the elapsed time from sending a request to receiving its first generated token. Measure from the client side with a monotonic clock, such as a high-resolution timer, and record the first actual content token—not merely the time headers arrive or the connection opens. Throughput usually means generated output tokens per second. State whether you mean aggregate throughput across requests or per-request generation speed; they answer different questions.
2. Choose a precise timing boundary. Start the timer immediately before the request is sent. For a streaming response, stop the TTFT timer when the client receives the first non-empty generated content. Stop the full-request timer when the response is complete. If the API sends role labels, metadata, or empty stream events before content, do not count those as the first token. Network delay is included in client-observed measurements, so note that these are end-to-end figures rather than model-only timings.
3. Count output tokens consistently. Prefer a server-reported completion-token count when it is available and trustworthy. Otherwise, tokenize the final generated text with the exact tokenizer and model configuration used by the deployment. Do not count prompt tokens as generated output. Record whether special tokens are included, and use the same rule for every run. Character or word counts are not substitutes for token counts.
4. Fix the test conditions. Select representative prompts, input lengths and requested output lengths. Keep sampling settings, model version, system prompt, tools, streaming mode and request parameters unchanged during a comparison. Note the deployment hardware or instance type, concurrency, software configuration and time of test when available. Avoid changing several variables at once: otherwise, a change in latency cannot be attributed clearly.
5. Run a baseline with one request at a time. Send enough requests to see normal variation rather than relying on one result. For each request, save TTFT, total elapsed time, output-token count, completion status and any error. Calculate per-request generation speed as output tokens divided by time from first token to completion. This excludes the initial wait and is useful for understanding decoding speed after generation starts.
6. Calculate aggregate throughput separately. For a batch or fixed measurement window, divide the total generated output tokens by the wall-clock duration of that same window. For example, if 6,000 output tokens are completed during a 30-second window, aggregate throughput is 200 tokens per second. Include all completed requests in the numerator and define exactly when the window starts and ends. Under concurrent load, this aggregate value can rise even while each user’s response becomes slower.
7. Test realistic concurrency gradually. Increase simultaneous requests in controlled steps, keeping the request mix comparable. At each level, record aggregate tokens per second, median and tail TTFT, errors, timeouts and per-request speed. Allow the deployment to reach a steady state before collecting each sample. Stop if error rates climb sharply, service limits are reached, or the test affects other users.
8. Summarize distributions, not just averages. Report sample count, median and a high percentile such as p95 for TTFT and total response time, alongside throughput. A few unusually slow requests can be hidden by a mean. Keep failed and timed-out requests visible as reliability results; excluding them silently makes performance look better than it is.
9. Repeat and compare fairly. Run the same test more than once, preferably at comparable load and with warm-up behavior documented. Compare matching percentiles and throughput at matching concurrency. Save the request mix and test configuration so a later model, server or hardware change can be evaluated against the same workload.

Common Mistakes

Do not call time to response headers TTFT: headers can arrive before generated text. Do not divide output tokens by total request time and label it decoding speed; that combines initial latency with generation. Conversely, do not use only the time after the first token when reporting user-perceived response latency.

Mixing prompt and completion tokens, using a different tokenizer, or comparing different output lengths makes throughput figures misleading. So does comparing a single-request test with a concurrent test. Report the workload and concurrency beside every number. Finally, avoid reporting only the best run: it hides normal variability and capacity-related slowdowns.

Tips

• Use a monotonic clock for elapsed-time calculations; wall-clock time can jump when the system clock changes.
• Keep raw per-request measurements, not just a chart or a single average. They let you recalculate percentiles consistently.
• Separate cold-start tests from steady-state tests. A newly started process, unloaded model or empty cache can have different latency from a warm deployment.
• Record streaming behavior and client network location. Client-observed TTFT includes transport and buffering effects.
• For very short outputs, throughput estimates are sensitive to timer precision and request overhead. Use longer, representative outputs and enough repeated requests.

Troubleshooting

TTFT is unexpectedly high: Check whether timing starts before DNS, connection setup or authentication; decide whether those belong in the end-to-end measure and keep the boundary consistent. Inspect server queueing, model loading, input length and network path. Confirm the client is registering the first content event rather than waiting for the stream to close.

Throughput seems implausibly low or high: Verify that the token count covers generated text only, that the timer uses seconds, and that numerator and time window cover the same requests. Check for buffering that delivers several chunks together, and use server-side token counts or logs as a cross-check. Confirm the benchmark has not included waiting time between separate test batches.

Results vary widely: Look for changing concurrency, mixed prompt sizes, cold starts, background traffic, retries and rate limiting. Repeat under a controlled workload, label warm-up samples, and report percentiles. If errors occur, retain them in the report and investigate service limits before raising load further.

Frequently Asked Questions

What is a good TTFT?
There is no universal target. It depends on the model, prompt length, deployment, network and user experience requirements. Establish a baseline for your workload and set a service objective that fits it.

Is TTFT the same as latency?
No. TTFT measures the initial wait for generated content. Total latency runs from request start until the response finishes. Both are useful and should be reported separately.

How do I calculate tokens per second for one request?
Divide generated output tokens by the elapsed time from the first content token to completion. State this boundary, since dividing by total request time gives a different measure.

Why can aggregate throughput improve while responses feel slower?
More concurrent requests may keep the hardware busier and increase total tokens produced per second, while queueing makes each request wait longer. Track TTFT and per-request speed alongside aggregate throughput.

Should I include prompt tokens in throughput?
For generated-token throughput, no. Report input or total token processing separately if that is useful, and clearly label the metric.

How many requests should I test?
Use enough repeated requests to estimate typical and tail behavior for your workload. There is no fixed count that suits every deployment; report the sample size and repeat tests under comparable conditions.

Related

• How to Test a Video Conferencing Room Before a Meeting
• Use AI Chatbots With Less Energy and Carbon
• Content Management: How to Organize Your Video Library