Benchmark On-Device AI Inference: NVIDIA vs AMD vs Intel

This guide walks you through benchmarking on-device AI inference across NVIDIA, AMD, and Intel GPUs. You’ll learn how to set up a consistent test environment, run standardized inference workloads, and compare performance metrics like latency and throughput. It’s aimed at developers and researchers who need reliable, reproducible numbers for model deployment decisions.

What You’ll Need

• A system with the GPUs you want to test (they can be in separate machines; results are comparable if the rest of the hardware is similar).
• Latest GPU drivers: NVIDIA (CUDA), AMD (ROCm), Intel (oneAPI or OpenVINO).
• Python 3.8+ with pip.
• PyTorch or TensorFlow installed for each GPU platform. For AMD, use the ROCm build; for Intel, use the XPU build (PyTorch) or Intel TensorFlow.
• ONNX Runtime (optional, for cross-platform benchmarking).
• A representative model: e.g., ResNet-50, MobileNetV2, or a small transformer like DistilBERT.
• Benchmarking script (we’ll write one) and a tool like time or a profiler.

Step-by-Step

1. Set up each GPU environment. For NVIDIA: install CUDA and cuDNN, then pip install torch torchvision --index-url https://download.pytorch.org/whl/cu121. For AMD: install ROCm, then pip install torch torchvision --index-url https://download.pytorch.org/whl/rocm5.6. For Intel: install Intel Extension for PyTorch (IPEX) and run with ipex.optimize(model).
2. Write a standard benchmarking script. Use the same model architecture, input shape, and batch size across all platforms. For example, load a pre-trained ResNet-50, create a random input tensor of shape (1,3,224,224), and run inference 100 times after a warm-up.
3. Include warm-up and timing. Run 10 iterations to warm up the GPU, then record the time for each of the next 100 iterations. Compute average latency (ms per inference) and throughput (inferences per second). Use torch.cuda.synchronize() for NVIDIA, torch.cuda.synchronize() works for ROCm too, and for Intel use torch.xpu.synchronize() if using XPU.
4. Run the script on each GPU. Execute the same script on each machine (or after switching GPUs). Save the output to a text file with the GPU name.
5. Repeat with different batch sizes. Test batch sizes 1, 8, 32, and 64 to see how the GPUs handle throughput. For each batch size, adjust the input tensor shape accordingly.
6. Test with different precisions. Run FP32 and FP16 (or BF16 for Intel) to compare performance. In PyTorch, use model.half() and cast input to half. For NVIDIA, you can also use TensorRT for optimized inference; for AMD, use ROCm’s MIGraphX; for Intel, use OpenVINO.
7. Use ONNX Runtime for cross-platform comparison. Export the model to ONNX, then run with ONNX Runtime using the appropriate execution provider: CUDA for NVIDIA, ROCm for AMD, OpenVINO for Intel. This gives a more apples-to-apples comparison.
8. Collect and normalize results. Record the average latency, throughput, and any power consumption if you have tools like nvidia-smi, rocm-smi, or intel_gpu_top. Normalize for any differences in CPU or RAM if using separate machines.
9. Analyze and report. Create a table with columns for GPU, precision, batch size, latency, and throughput. Highlight which GPU is best for low-latency vs high-throughput scenarios.

Common Mistakes

• Not using the same model version. Ensure the exact same model weights and architecture are used. A different version can change results.
• Forgetting warm-up. GPUs often run slower initially due to lazy initialization. Always warm up before timing.
• Ignoring CPU bottleneck. If the CPU is slow at preprocessing or data loading, it can skew results. Use pre-generated tensors in memory.
• Mixing precision. Comparing FP32 on one GPU and FP16 on another without noting it is invalid.
• Not syncing. On CUDA, timing without torch.cuda.synchronize() captures kernel launch time, not execution time.

Tips

• Use a fixed random seed to ensure identical input data across runs.
• Run each test multiple times (e.g., 3 runs) and take the median to reduce variance.
• For Intel GPUs, enable the oneDNN backend for PyTorch: torch.backends.mkldnn.enabled = True.
• If you have access to a cluster, use the same CPU and RAM configuration to isolate GPU performance.
• Consider using a profiling tool like NVIDIA Nsight, AMD ROCm Profiler, or Intel VTune for deeper insights.

Troubleshooting

• CUDA out of memory: Reduce batch size or use a smaller model. Check with nvidia-smi for memory usage.
• ROCm not detecting GPU: Run rocm-smi to verify. Install the correct ROCm version for your GPU architecture.
• Intel GPU not recognized: Ensure the latest Intel graphics driver is installed and that your PyTorch build supports XPU. Try torch.xpu.is_available().
• ONNX Runtime fails to load execution provider: Install the correct ONNX Runtime package: onnxruntime-gpu for CUDA, onnxruntime-rocm for AMD, onnxruntime-openvino for Intel.
• Inconsistent results between runs: Check for thermal throttling; ensure the GPU is not overheating. Run with a cool system.

Frequently Asked Questions

Can I benchmark all three GPUs on one machine?

Yes, if you have a motherboard with multiple PCIe slots and a power supply that can handle all GPUs. However, you must ensure each GPU has its own driver stack and that the system is stable.

Is it fair to compare different GPU architectures?

It’s fair if you’re measuring real-world performance for your workload. But note that some architectures may be optimized for specific operations, so results may vary by model.

Why is my NVIDIA GPU faster in PyTorch but slower in ONNX Runtime?

PyTorch uses its own CUDA kernels, while ONNX Runtime’s CUDA provider may be less optimized for your model. Conversely, OpenVINO might be faster for Intel on ONNX.

Do I need to use the same batch size for all GPUs?

Yes, to compare throughput directly. But you should also test each GPU’s optimal batch size to see its maximum throughput.

What is the best metric to compare?

It depends on your use case. For real-time applications, latency is key. For offline processing, throughput (inferences per second) matters more.

Can I use TensorRT, MIGraphX, and OpenVINO for a more optimized comparison?

Yes, that’s a good idea if you’re deploying in production. These tools optimize the model for each specific GPU, giving a more realistic performance estimate.

Related

• Huawei Ban: US Technology Dependence Explained
• Never Buying Another Ink Cartridge Printer?
• Jobs Least Exposed to AI Automation