Benchmark an AI Serving Harness Without Misleading Results

This guide walks you through benchmarking an AI serving harness—the system that routes requests to your models and returns responses—so you get numbers you can trust. You’ll learn how to design a fair test, avoid common pitfalls that inflate or deflate results, and interpret what the metrics really mean. Before starting, you’ll need a running harness, a set of representative prompts, and a way to record timestamps and resource usage.

What You’ll Need

• A deployed AI serving harness (e.g., vLLM, Triton, or a custom wrapper) with at least one model loaded.
• A representative dataset of prompts—ideally a few hundred to a few thousand, matching your real usage patterns.
• A load testing tool like wrk, hey, or a custom script that can send concurrent requests.
• System monitoring tools (e.g., top, nvidia-smi, dstat) to track CPU, GPU, memory, and network.
• A controlled environment—no other heavy workloads running on the same hosts.

Step-by-Step

1. Define your goal and metrics. Decide what you’re optimizing for: latency (time to first token or full response), throughput (requests per second), or both. Choose primary metrics like p50, p95, and p99 latency, plus requests per second. Avoid relying on averages alone—they hide outliers.
2. Prepare a representative workload. Use prompts that match your production traffic in length, topic, and expected output tokens. If your users ask short questions, don’t benchmark with 500-word essays. Include a mix of input lengths and output lengths, as these dramatically affect performance.
3. Warm up the system. Before measuring, send a few hundred requests to let the model load, caches fill, and any adaptive components stabilize. Otherwise, your first measurements will include cold-start latency.
4. Run a baseline at low concurrency. Start with a single user (concurrency=1) to measure the best-case latency. Record the time to first token (TTFT) and total response time. This gives you a ceiling for what the system can do with no contention.
5. Increase concurrency gradually. Double the number of concurrent requests (e.g., 1, 2, 4, 8, 16, 32) and run each level for a fixed duration (e.g., 2–5 minutes) to allow steady-state behavior. Record latency percentiles and throughput at each level.
6. Monitor system resources. While the test runs, capture CPU, GPU utilization, memory, and network I/O. This tells you whether you’re bottlenecked on compute, memory, or I/O. For GPU-based serving, watch for GPU memory exhaustion, which can cause errors or swapping.
7. Repeat each test at least 3 times. Variability is normal; run each concurrency level multiple times and report the median or mean of the runs. If results vary wildly, investigate the cause (e.g., background processes, network jitter).
8. Record everything. Log the exact model version, harness version, batch size, max tokens, temperature, and any quantization settings. Without this, your results are not reproducible.
9. Analyze and report. Plot latency percentiles vs. concurrency, and throughput vs. concurrency. Identify the ‘knee’ where latency spikes or throughput plateaus—that’s your practical capacity. Present results with confidence intervals if possible.

Common Mistakes

• Using average latency only. Averages hide tail latency. A system could have a 50ms average but p99 of 2 seconds—unacceptable for real users. Always report percentiles.
• Benchmarking with a single short prompt. Real workloads vary; a single prompt doesn’t stress the system realistically. It may fit in cache and give unrealistically fast results.
• Not warming up. Cold starts can add seconds to the first requests, skewing your results. Always warm up.
• Ignoring output token count. If your benchmark prompts generate 10 tokens but production generates 500, your throughput will be misleadingly high. Output length is a major factor.
• Running tests on a shared machine. Other processes can steal CPU or GPU, making results noisy. Use a dedicated environment.
• Changing parameters mid-test. If you tweak batch size or max tokens between runs, you can’t compare results. Keep everything constant except the variable you’re testing.

Tips

• Use a fixed random seed for prompt generation to ensure identical inputs across runs.
• Measure time to first token (TTFT) separately from total response time. TTFT is critical for interactive applications.
• Test with different input lengths (short, medium, long) and report results per category—they often behave differently.
• Consider using a tool like locust for more realistic load patterns with ramping users.
• Record the exact model and harness versions in your report; performance changes with updates.
• If your harness supports continuous batching, test with and without it to see its impact.

Troubleshooting

• Results are wildly inconsistent. Check for background processes, network congestion, or thermal throttling. Run each test multiple times and look for outliers.
• Throughput drops at high concurrency. This could indicate resource exhaustion (GPU memory, CPU) or queueing delays. Monitor system metrics to confirm.
• Errors appear (timeouts, 500s). Your harness may have a max concurrency limit. Check logs and adjust the load or the harness configuration.
• Latency is high even at concurrency=1. The model might be too large for the hardware, or the harness has overhead. Try a smaller model or optimize the harness settings.
• GPU memory usage is near 100%. Reduce batch size or use a smaller model. Memory pressure can cause swapping and severe latency spikes.
• Results differ between runs. Ensure you’re using the same dataset and parameters. If using a cloud environment, instance types may vary—use the same instance type.

Frequently Asked Questions

• How many requests should I send? Enough to reach steady state—typically at least a few hundred, but for reliable percentiles, aim for 1000+ requests per test.
• What is a good p99 latency for an AI service? It depends on your use case. For interactive chat, under 1 second is often desired; for batch processing, longer is acceptable. Set your own SLO based on user expectations.
• Should I benchmark with real user traffic or synthetic? Real traffic is ideal but often hard to capture. Synthetic that mimics your traffic patterns in length and frequency is a good substitute.
• How long should each test run? At least 2–3 minutes after warm-up to allow the system to stabilize. Longer tests (10+ minutes) can reveal memory leaks or degradation.
• Why does my throughput not scale linearly with concurrency? Because of contention for shared resources (GPU, CPU, memory). At some point, adding concurrency increases queueing delay without increasing throughput—that’s your saturation point.
• Can I benchmark on a laptop? You can for basic testing, but results won’t reflect production hardware. Use a representative server or cloud instance for meaningful numbers.

Related

• Docker Compose Explained: Services, Volumes, Networks
• Fix TWRP Unable to Mount Storage 0MB
• Sea Ice vs Glaciers: Which Melting Raises Sea Level?