Measure AI Agent Speed, Tokens, and Production Cost

Measure an AI agent’s speed, token use, and cost by recording each model call alongside the full request’s start and end times. This gives you comparable production metrics, helps reveal which steps drive latency or spend, and makes it easier to spot regressions. Before you begin, make sure you can add instrumentation to the agent service and securely view its logs or metrics.

What You’ll Need

• Access to the agent’s application code, deployment settings, or observability platform.
• A way to emit structured logs or metrics, such as your existing monitoring stack.
• Provider usage data for each model call, including input and output tokens where available.
• A reliable way to associate calls with a request or trace ID without logging sensitive content.
• Current model pricing from the provider, recorded with the currency and effective date.

Step-by-Step

1. Choose the measurements and their boundaries. Track at least end-to-end duration, model-call duration, input tokens, output tokens, and estimated model cost. Define end-to-end duration as the time from accepting a user request until the agent has produced its final response. Decide whether the measurement includes queueing, tool execution, retries, and post-processing; record those separately rather than silently changing the definition. For interactive systems, also measure time to first response or first streamed token if users see output before completion.
2. Set up a trace for each user request. Generate a unique, non-sensitive trace ID when work begins and pass it through agent steps, model calls, tools, and retries. Give each model call its own span or structured record, linked to the parent request. Include a deployment or service version, model identifier, timestamp, and outcome. Avoid putting prompts, completions, credentials, or personal data in metric labels; use access-controlled logs only when content is genuinely needed and permitted.
3. Measure elapsed time with a monotonic clock. Capture a start time immediately before the operation and an end time immediately after it. Calculate elapsed milliseconds using a monotonic timer, not subtraction of wall-clock timestamps: system clock adjustments can make wall-clock durations inaccurate. Record separate spans for queue wait, model request, tool execution, and other meaningful stages. For streaming, capture both time to first token and total generation time. Do not compare durations until you have confirmed that all services use the same boundary definitions.
4. Capture tokens from the response’s usage fields. Read the provider’s usage metadata for every completed model call and save input and output token counts separately. If the response reports cached input, reasoning, or other token categories, retain those fields too and follow the provider’s billing definitions. Store the model and API mode alongside the counts. Do not treat a local tokenizer estimate as billed usage; use estimates only when the response has no usage data, and mark them clearly as estimates.
5. Record retries and agent steps, not just totals. An agent request may make several model calls. Save each call’s duration and token counts, along with its step name, attempt number, and success or failure status. Count failed calls and retries because they may consume tokens and time even if the user sees only one final response. Use the trace ID to aggregate calls into request-level totals, while retaining call-level records to identify expensive or slow steps.
6. Calculate cost using the applicable rate card. For a model with separate input and output rates, calculate estimated cost as (input tokens × input price per token) + (output tokens × output price per token). If prices are quoted per million tokens, divide each token count by 1,000,000 before multiplying by its per-million rate. Apply separate rates for cached tokens or other categories when the provider specifies them. Keep the rate, currency, pricing date, and model with the calculation. Label the result an estimate unless it comes from the provider’s final billing records; other charges may not be included.
7. Build dashboards around distributions and cohorts. Display request volume, p50 and p95 end-to-end latency, time to first token where relevant, tokens per request, and estimated cost per request. Add model, environment, outcome, and agent version as carefully bounded dimensions. Break down latency and cost by model call or step. Percentiles show whether a typical request differs from a slow tail; an average alone can hide occasional long waits. Compare like with like, such as the same task type and model configuration.
8. Validate the instrumentation before relying on it. Run a small controlled set of requests and compare recorded usage with the response metadata and provider usage dashboard. Check that a multi-step request sums all its calls, that retries appear, and that durations match a local trace. Verify cost calculations against a hand-worked example using the recorded rate. Then monitor missing fields, duplicate events, clock or unit mix-ups, and differences between estimated totals and billing data.
9. Set alerts and review changes safely. Alert on meaningful increases in p95 latency, token use, error rate, or cost per request, using a baseline for the same workload. Avoid paging on a single unusually expensive request unless it crosses a deliberate threshold. When changing a prompt, model, tool, or retry policy, compare a representative sample before and after deployment. Keep the metric definitions and pricing assumptions visible so later comparisons remain interpretable.

Common Mistakes

• Timing only the model call: Users experience queueing, tools, retries, and orchestration too. Track both request-level and per-stage durations.
• Adding tokens from only the final call: Aggregate every call in the trace, including retries and calls that fail after consuming usage.
• Mixing milliseconds and seconds: Name units in field names or schemas, and convert consistently before calculating percentiles.
• Using one token price for all usage: Input, output, cached, and other categories can have different rates. Confirm the current rate card for the exact model.
• Calling estimates exact bills: Provider billing can include categories or adjustments absent from application telemetry. Reconcile estimates with provider records.
• Adding high-cardinality or sensitive labels: Request IDs, user IDs, and prompt text can increase monitoring costs or expose information. Keep identifiers in controlled traces, not broad metric dimensions.

Tips

• Use one request ID across services and a separate span ID for each operation.
• Store raw counts and durations; calculate rollups in dashboards so you can revise views without losing detail.
• Track both tokens per request and tokens per successful response. The distinction can expose waste from failures and retries.
• Record the model alias and resolved model where available, since aliases and deployments can change.
• Set retention and access controls appropriate to operational telemetry, and avoid collecting conversation content by default.

Troubleshooting

Token counts are missing: Check whether the API response includes usage data for that endpoint and whether your streaming client captures the final usage event. If it does not, use provider reporting for reconciliation and mark any tokenizer-derived figures as estimates.

Dashboard cost differs from the bill: Confirm the pricing date, model, currency, token categories, and per-million conversion. Check for retries, cached input, batch or other pricing modes, and provider-side usage not represented in your service. Treat billing records as authoritative for billed totals.

Latency looks unusually high or negative: Confirm that durations use a monotonic clock and consistent units. Inspect queueing, network timeouts, tool spans, and retry attempts separately. A negative elapsed value often points to wall-clock subtraction or a bad timestamp conversion.

Requests appear duplicated or incomplete: Check whether events are emitted both before and after retries, whether asynchronous work loses its trace context, and whether a timeout prevents the final record from being written. Make event handling idempotent where possible and record a clear terminal status.

Frequently Asked Questions

What is the difference between latency and throughput?
Latency is the time one request takes. Throughput is how many requests or tasks the system completes over a period.

Should I measure tokens per agent step or per request?
Measure both. Per-step data finds costly operations; request totals reflect the full workload and support cost-per-request comparisons.

How do I measure streaming speed?
Record time to first token and total response duration. You can also calculate generation rate from output tokens divided by generation time, but state which timing interval you use.

Can token counts predict the exact production bill?
Not always. They support an estimate, but pricing categories, retries, billing adjustments, and other usage may affect the final amount. Reconcile against provider usage reports.

How often should I review these metrics?
Keep dashboards current and alert on sustained or significant changes. Review trends after deployments and reconcile cost estimates with billing reports on a regular operational cycle.

Do I need to log prompts to measure performance?
No. Durations, usage counts, model identifiers, outcomes, and trace relationships are usually enough. Avoid storing prompt or response content unless a specific, approved debugging need requires it.

Related

• Track Renewable Energy Certificates From a Solar PPA
• MCP for AI Coding Assistants and Video Generation
• How to Get the Brick Rod in Fisch