LLM Profiling on CUDA or ROCm: Optimize Step by Step

Profiling an LLM on CUDA or ROCm helps you find whether time is going to model computation, memory movement, kernels, or system overhead before changing code. This guide walks through a repeatable baseline-to-optimization process for inference or training, using the profiler and tools available in your framework.

What You’ll Need

• A supported GPU, working driver, and compatible CUDA or ROCm runtime.
• A framework build that matches your GPU runtime, plus the model and a reproducible workload.
• Enough disk space for traces and permission to use the GPU profiling tools. Some system-level counters require additional permissions.
• A way to record the baseline: GPU model, software versions, model settings, input lengths, batch size, and measured latency or throughput.

Step-by-Step

1. Confirm the stack before measuring. Record the GPU, driver, framework version, and runtime build. On NVIDIA, check that the framework reports CUDA and sees the device; on AMD, check that it reports ROCm/HIP and sees the device. Use the framework’s own device query or a basic tensor operation. A visible GPU does not prove that every kernel or profiler feature is supported.
2. Define a representative workload. Decide whether you are profiling inference or training, then hold the model, precision, batch size, sequence lengths, and generation settings constant. For inference, measure prompt processing (prefill) separately from token generation (decode): prefill processes many tokens in parallel, while decode often generates one token at a time. For training, include forward, backward, and optimizer steps. Do not use a tiny toy input as the only benchmark.
3. Warm up and establish a clean baseline. Run several untimed iterations to load weights, initialize kernels, and allow compilation or autotuning to finish. Then run enough timed iterations to smooth out noise. Synchronize the device around measurements when using host-side timers; GPU work is asynchronous, so timing only the launch can understate elapsed time. Report median and a spread such as p90, not just the best run. Keep other GPU jobs idle and note power or thermal constraints.
4. Measure the outcome that matters. For serving, capture end-to-end latency, time to first token, inter-token latency, and tokens per second; state whether throughput is per request or aggregate. For training, capture step time and examples or tokens per second. Also record peak allocated and reserved memory if relevant. Use the same measurement boundaries for every comparison.
5. Start with a framework-level profile. Use PyTorch Profiler or the equivalent profiler for your framework to capture CPU and GPU activities, operator shapes, memory events, and a short representative window. Keep the capture short: profiling every iteration can add substantial overhead and generate huge traces. Inspect the timeline for gaps, synchronization, repeated copies, CPU-bound launches, and dominant GPU operators. Separate setup and compilation from steady-state work.
6. Use the vendor timeline when the framework trace leaves questions. NVIDIA users can capture a short timeline with Nsight Systems and inspect selected kernels with Nsight Compute. AMD users can use ROCm profiling tools such as rocprofiler and the ROCm systems trace tooling available for their release. Tool names, permissions, and supported counters vary by version. Check that the installed profiler supports the GPU and runtime; do not infer performance from a trace captured with unsupported features.
7. Classify the bottleneck before changing anything. Long GPU kernels with high utilization suggest compute or memory-bound work; frequent short kernels and launch gaps may point to launch overhead or CPU scheduling. Large memory use or transfers can indicate bandwidth pressure, activation storage, or inefficient movement. For LLMs, decode may be limited by weight or key-value-cache bandwidth, while prefill can be compute-heavy. Confirm with more than one signal; utilization alone is not a diagnosis.
8. Change one variable at a time. Choose a change that addresses the observed bottleneck: test a supported lower-precision format, adjust batch size, enable an appropriate fused attention/kernel path, reduce unnecessary padding, or improve data loading for training. For serving, consider batching only with latency and memory constraints in view. Avoid assuming a feature is available simply because it exists in another CUDA or ROCm release.
9. Re-run the same benchmark and validate correctness. Compare against the baseline using identical inputs, warm-up, timing boundaries, and system conditions. Check output quality or numerical tolerances, not just speed. Record the change, software settings, and results. Keep an optimization only when it produces a repeatable benefit for the target workload without unacceptable memory use or correctness regressions.

Common Mistakes

• Timing asynchronous GPU launches without synchronization, or including model loading in one run but not another.
• Mixing prompt lengths, batch sizes, precision, or generation settings between measurements.
• Profiling a long run and treating profiler overhead as normal performance.
• Optimizing a single kernel while ignoring end-to-end latency, CPU preprocessing, or communication.
• Assuming CUDA and ROCm have identical kernel availability, environment variables, profiler features, or performance behavior.
• Comparing a cold first run with a warmed-up run, or reporting only the fastest result.

Tips

• Keep a small benchmark script and configuration file under version control so each result is reproducible.
• Profile a short representative interval after warm-up, then use ordinary timing runs for final comparisons.
• For variable-length requests, report the length distribution and separate prefill from decode wherever possible.
• Track memory headroom as well as speed. A larger batch can improve throughput but trigger out-of-memory errors or worsen tail latency.
• Save traces with concise names that include hardware, workload, and change; traces may contain sensitive workload details, so store them appropriately.

Troubleshooting

The GPU is not detected: verify the driver and runtime installation, then check that the framework build targets the installed CUDA or ROCm stack. Confirm device visibility outside the application and inspect startup logs. Avoid mixing incompatible libraries from different environments.

Results vary widely: stop competing GPU workloads, warm up consistently, synchronize measurements, and increase the number of timed repetitions. Check for changing sequence lengths, dynamic batching, compilation, thermal limits, or power-management changes.

The trace is empty or incomplete: confirm the profiler’s supported GPU/runtime combination and that GPU activities were enabled. Try a short capture around known work, and check permissions required for hardware counters. Framework-level traces can still help when low-level counters are unavailable.

Profiling slows the model dramatically: reduce the capture window and disable unnecessary stack traces or expensive counters. Use an unprofiled benchmark for final latency and throughput numbers.

An optimization makes results worse or fails: revert the single change, verify its prerequisites, and inspect memory use and correctness. Kernel selection may depend on shapes, data type, GPU architecture, or runtime version; benchmark the actual production-like shapes rather than assuming a universal win.

Frequently Asked Questions

Should I use a framework profiler or a vendor profiler first?
Start with the framework profiler to locate operators and timeline gaps. Move to a vendor tool when you need lower-level kernel, scheduling, or hardware-counter detail.

Can I compare CUDA and ROCm results directly?
Yes, if the model, inputs, precision, workload, timing method, and hardware conditions are clearly stated. Different toolchains and kernel implementations mean the comparison measures complete software-and-hardware configurations, not just runtimes.

How long should a profiling capture be?
Capture only enough warmed-up work to show a repeatable pattern—often a few steps or a short sequence of requests. Keep a separate, longer unprofiled run for stable performance statistics.

Why is GPU utilization low while inference is slow?
Possible causes include CPU launch overhead, synchronization, small batches, data movement, or decode’s sequential workload. Inspect the timeline and separate prefill from decode before changing batch size or kernels.

Does mixed precision always improve LLM speed?
No. Support and benefit depend on the GPU, model operations, framework, and runtime. Validate numerical behavior, memory use, and end-to-end performance on the target workload.

Related

• Intelligence, IQ and AI: Why Ability Is a Spectrum
• Panasonic NN-ST64JWQPQ: Power, Features and Fit
• Data Recovery Trends: What’s Changing and Why