keepcalculator

06 Numbers and Powers

AI Gateway Latency: Find the Bottleneck

Card No. 06·84334 · Updated · Standard formula, shown below

If your AI gateway is getting slower, the delay may come from the gateway itself, the model provider, network connections, or work your application performs before and after a request. Find the bottleneck by measuring each stage separately, comparing like-for-like requests, and checking whether the change affects all traffic or only particular models, routes, or payloads.

How an AI gateway adds to request time

An AI gateway typically sits between an application and one or more model services. Depending on its configuration, it may authenticate requests, enforce limits, select a provider, transform messages, apply safety checks, retry failures, and record logs or traces. Each step can add time, but total user-perceived latency also includes the model’s processing and the journey back to the application.

For a non-streaming request, the user usually waits for the response to finish before seeing it. With streaming, the first visible text may arrive much earlier than the completed response. These are different measurements: time to first token (TTFT) describes how long until the first generated token arrives, while total response time includes the remaining generation and delivery. A gateway can therefore feel slow even if its total duration is reasonable, or appear fast by TTFT while taking a long time to finish.

Think of elapsed time as a sequence of stages rather than one unexplained number:

  • Application work before sending the request, such as building a large prompt.
  • DNS lookup, connection setup, and network travel to the gateway.
  • Gateway processing, routing, queueing, and any configured checks.
  • Provider queueing and model inference, including prompt processing and token generation.
  • Response transfer, gateway processing, and application work after receipt.

Some stages overlap, especially when streaming, so adding independently measured durations is not always exact. The sequence is still a useful map for deciding what to measure.

Measure the stages before changing settings

Start with timestamps from the client, gateway, and provider where available. Record a request identifier so that events can be matched across systems, and use synchronized clocks if you are comparing timestamps from different machines. Prefer gateway-provided durations for internal stages; clock differences can make cross-system subtraction misleading.

For each request, collect a small set of comparable dimensions: route or provider, model, streaming mode, prompt size, output size, status, retry count, and timestamps for request arrival, upstream dispatch, first upstream response, and completion. Avoid logging full prompts or responses unless there is a clear, controlled need; they can contain sensitive information.

Compare distributions, not only averages. Median latency describes a typical request, while high percentiles such as p95 show slower experiences that averages can conceal. Break results down by route and request characteristics. If one model’s p95 rises while other routes remain steady, a gateway-wide average may obscure the real pattern.

A practical bottleneck-finding sequence

  1. Confirm the change. Compare the same routes and request types before and after the slowdown. Check whether the metric is TTFT, total duration, or both, and whether a traffic increase or larger prompts explains the difference.
  2. Separate gateway time from upstream time. Look at gateway processing duration and upstream request duration. If the gateway spends more time before dispatch, inspect its queue, routing, authentication, policy checks, and resource use. If the upstream interval grows, investigate provider performance, model choice, network path, and provider-side queueing.
  3. Check timing by phase. A longer wait before the first response points toward connection establishment, queueing, prompt processing, or provider response time. A normal TTFT but longer completion often points toward more generated tokens, slower token generation, or response transfer.
  4. Inspect retries and timeouts. A request that fails once and succeeds on retry can take much longer than a successful first attempt. Check retry counts, backoff delays, timeout settings, and provider errors. Retries can also increase load during an outage, so do not simply increase them without considering the effect.
  5. Run a controlled comparison. Send similarly sized requests through the same route and, where permitted, compare with a direct provider request. Keep model, prompt, output limit, region, and streaming behavior consistent. A direct test is diagnostic, not proof: it may take a different network path or bypass gateway policies.
  6. Change one factor at a time. Test a smaller prompt, a different route, or a reduced output limit in a controlled setting. If several settings change together, it becomes difficult to identify which one mattered.

Common causes and what the evidence suggests

  • Large prompts or long outputs: More input can increase prompt-processing time, and more generated tokens generally take longer to deliver. Compare requests with similar token counts rather than comparing unrelated conversations.
  • Provider or model variation: Different models and provider routes can have different response times. A model change, fallback route, or provider incident can raise latency without any gateway software change.
  • Queueing and capacity pressure: Rising traffic, limited concurrency, or overloaded gateway resources can create waiting time. Check queue depth, CPU, memory, connection pools, and request volume together; a high resource reading alone does not establish the cause.
  • Retries and fallback routing: A slow or unavailable first provider may trigger another attempt. Trace the full route history instead of looking only at the final successful provider.
  • Connection and network behavior: Repeated DNS lookups, new TLS connections, packet loss, or a distant region can add delay. Compare reused and newly established connections where your instrumentation allows it.
  • Gateway features and observability: Content transformations, policy checks, logging exporters, or synchronous calls to other services may add work. Measure them before disabling controls; removing a safety or governance feature can create risks without fixing the actual bottleneck.

Interpret results without jumping to conclusions

A high gateway duration does not automatically mean the gateway’s core code is slow. It may include time spent waiting for an upstream response, depending on how the dashboard defines the metric. Read metric definitions and traces before attributing delay. Likewise, a healthy average does not rule out a serious tail-latency problem affecting a small share of requests.

Streaming is another common source of confusion. Measure TTFT and completion time separately, and check whether the client displays chunks as they arrive. A slow-looking interface may be buffering output even when the gateway streams correctly. Conversely, a prompt response with a long gap before the first token may indicate queueing or upstream processing, not a rendering problem.

Make performance changes only after locating the delay. Possible remedies include reducing unnecessary prompt content, choosing a suitable model or route, improving connection reuse, addressing gateway capacity, or removing redundant processing. Validate the result with the same request profile and watch error rates and high-percentile latency as well as the average.

Frequently Asked Questions

Why is my AI gateway slow if the provider status looks normal?
A provider’s general status does not describe every model, region, route, or individual request. The delay may also be in your network, gateway queue, retries, prompt processing, or application. Compare phase-level timings for affected requests.

What is the difference between TTFT and total latency?
TTFT is the time from sending a request until the first generated token arrives. Total latency is the time until the complete response is received. Streaming can improve TTFT without reducing total generation time.

Can a larger prompt make responses slower?
Yes. A larger input can take longer for a model to process. Longer requested outputs can also take more time because more tokens must be generated and transferred.

Should I bypass the gateway to test latency?
A controlled direct request can help distinguish gateway-side delay from upstream delay, if your policies permit it. Keep the request and route comparable, and remember that bypassing the gateway changes the network path and skips its configured controls.

Do retries improve or worsen latency?
Retries can recover from transient failures, but they add time and may increase load. Check retry counts and backoff intervals, then tune them against both reliability and latency goals rather than increasing retries by default.

Related

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.

More cards in this drawer

The whole Numbers and Powers drawer →