Benchmark AI Inference Hardware for Your Workload

This guide walks you through benchmarking AI inference hardware—GPUs, CPUs, or specialized accelerators—so you can pick the right machine for your specific model and traffic. You’ll define a realistic workload, set up a measurement harness, run controlled tests, and interpret the results. No prior benchmarking experience is needed, but you should be comfortable with the command line and have a model you want to test.

What You’ll Need

• A machine with the hardware you want to test (local or cloud instance).
• Your trained model in a deployable format (e.g., ONNX, TensorRT, PyTorch, TensorFlow).
• Python 3.8+ and pip installed.
• Inference framework(s) matching your model (e.g., PyTorch, TensorFlow, ONNX Runtime).
• GPU drivers and CUDA/cuDNN if testing NVIDIA GPUs (or ROCm for AMD).
• A tool to generate load: hey, wrk, locust, or a simple Python script.
• Monitoring tools: nvidia-smi, htop, or cloud provider metrics.

Step-by-Step

1. Define your workload. Write down the exact input shape (e.g., 224×224 RGB images), batch size (e.g., 1, 8, 32), and data type (FP32, FP16, INT8). Also specify the expected request rate and latency target (e.g., p95 under 50 ms). This is the contract your benchmark will measure against.
2. Prepare the model. Export your model to a format optimized for inference. For PyTorch, use torch.jit.script or torch.onnx.export. For TensorFlow, freeze the graph or use TF-TRT. If using TensorRT, build an engine with trtexec or the Python API. Save the optimized model to disk.
3. Set up a clean environment. Create a fresh Python virtual environment on the target machine. Install the inference framework, the model’s preprocessing dependencies, and a benchmarking library like onnxruntime or torch. Avoid installing training libraries that may slow things down.
4. Write a minimal inference script. Create a Python script that loads the model, generates a random input tensor of the correct shape, runs inference once to warm up, then runs a loop of N iterations (e.g., 1000) and records the time per iteration. Use time.perf_counter() for precision. Include a function to preprocess real data if your workload requires it.
5. Measure single‑request latency. Run the script with batch size 1 and a small N (e.g., 100) to get the average and percentile latencies. Record the p50, p95, and p99. This tells you the raw speed of the hardware for one request.
6. Measure throughput under load. For a more realistic test, use a load generator to send concurrent requests to a serving endpoint. If you have a serving framework (e.g., TorchServe, Triton), start it and point hey or wrk at it. Alternatively, write a threaded Python script that sends requests to your model in a loop. Increase concurrency gradually (1, 4, 8, 16, 32) and record the requests per second (RPS) and latency percentiles at each level.
7. Monitor hardware utilization. While the load test runs, sample GPU utilization, memory usage, and CPU usage. On NVIDIA GPUs, use nvidia-smi dmon -s puc to log utilization and power. On CPU, use top or htop. Save these logs for analysis.
8. Repeat with different batch sizes. Change the batch size in your inference script (e.g., 1, 8, 32, 64) and rerun the latency and throughput tests. Larger batches often increase throughput but hurt latency. Find the sweet spot for your workload.
9. Test with your real data distribution. If possible, use a sample of your actual input data instead of random tensors. Some models behave differently on real data due to memory access patterns or branch prediction. Preprocess the data exactly as you would in production.
10. Compare across hardware. Repeat the same steps on each candidate machine (e.g., A100 vs. T4 vs. CPU). Keep the software versions identical across tests to ensure a fair comparison. Record all results in a spreadsheet with columns: hardware, batch size, concurrency, p50, p95, p99, RPS, GPU util, power.

Common Mistakes

• Not warming up. The first few inferences include lazy initialization and kernel compilation. Always run a few warm‑up iterations before timing.
• Mixing data types. Comparing FP16 on one GPU to FP32 on another is not apples‑to‑apples. Decide on a data type and stick to it.
• Ignoring CPU bottlenecks. If the GPU is idle but the CPU is pegged, your benchmark is measuring CPU preprocessing, not GPU inference. Profile to see where the time goes.
• Using too few iterations. With 10 iterations, noise dominates. Use at least 100, and more if the latency is high.
• Forgetting to pin CPU cores. On multi‑socket servers, thread migration can cause inconsistent results. Use taskset to pin your process to specific cores.

Tips

• Run each test at least three times and take the median to reduce variance.
• Set a fixed random seed for input generation to make tests reproducible.
• For cloud instances, use the same instance type and region for all tests to avoid network variability.
• If you’re benchmarking a serving stack, include the full pipeline: preprocessing, network, and postprocessing. That’s what users experience.
• Use a tool like perf on Linux to get CPU instruction counts if you suspect CPU bottlenecks.
• Record the exact software versions (CUDA, cuDNN, framework) in your results. A one‑line version difference can change performance by 20%.

Troubleshooting

• Results are wildly inconsistent. Check for thermal throttling. On GPUs, run nvidia-smi to see if the temperature is near the limit. On CPUs, check sensors. Also disable any background processes (e.g., cron jobs, cloud agent updates).
• GPU utilization is low (under 50%). Your workload is likely bottlenecked by data loading or preprocessing. Profile your script with py-spy or cProfile. Move preprocessing to a separate thread or use a faster data loader.
• Out of memory. Reduce the batch size or use a smaller input resolution. For GPUs, check nvidia-smi for memory usage. If using TensorRT, rebuild the engine with a smaller workspace size.
• Latency spikes at high concurrency. This is normal as queues fill up. Look at the p99 and p95—if they are far above the p50, your system is not designed for that concurrency level. Reduce concurrency or add more replicas.
• Different results on different runs. Ensure you’re using the same input data. If you generate random inputs each run, the model may have variable‑length loops or memory allocations. Save a fixed input tensor to disk and load it each time.

Frequently Asked Questions

Q: How long should each benchmark run?
A: At least 1–2 minutes per configuration after warm‑up. Longer runs (5–10 minutes) are better for detecting thermal throttling or memory leaks.

Q: Should I use a batch size of 1 for latency?
A: Yes, if your production traffic is single requests. But also test the batch size you plan to use, because some hardware has high fixed overhead that amortizes better with larger batches.

Q: What is the best metric to compare hardware?
A: It depends on your goal. For real‑time applications, look at p95 latency. For offline processing, throughput (requests per second) matters more. Always report both.

Q: Can I benchmark without a serving framework?
A: Yes, a simple Python loop that calls the model directly is fine for raw hardware comparison. But for production decisions, include the serving overhead.

Q: How do I handle models with dynamic input shapes?
A: Fix the input shape for benchmarking. Dynamic shapes often trigger re‑compilation and make results noisy. If you must test dynamic shapes, use a representative distribution of sizes and report the median.

Q: Is FP16 always faster than FP32?
A: Not always. On some GPUs, FP16 can be slower if the model has operations that fall back to FP32. Test both and measure accuracy loss as well.

Related

• 4K Monitors High Refresh Rates: What to Know
• Kodi TV Boxes: What They Are & How They Work
• How Wordle Chooses Its Daily Answer (Not Random)