AI data centers are hitting a performance wall where adding more GPUs often does not speed up large model training or inference, because the real bottleneck is how fast data can move between the processor and memory. The solution is not simply scaling the number of accelerators, but increasing the memory bandwidth per GPU and across the system. This article explains why memory bandwidth has become the critical constraint in AI infrastructure, how it affects performance, and what architects are doing about it.
Formula
The calculator above estimates the effective memory bandwidth available per GPU when you have a certain total bandwidth per GPU and a number of GPUs. The formula is:
Effective bandwidth per GPU (GB/s) = (Total bandwidth per GPU (TB/s) × 1000) / Number of GPUs
Variables:
- Total bandwidth per GPU (TB/s): the theoretical peak memory bandwidth of a single GPU, as specified by the manufacturer.
- Number of GPUs: the total number of GPUs in the cluster.
- Effective bandwidth per GPU (GB/s): the average memory bandwidth each GPU can actually use when the system is fully loaded, assuming no other bottlenecks.
Worked Example
Using the default values: a single GPU has 4 TB/s memory bandwidth, and you have 1,000 GPUs. The effective bandwidth per GPU is:
(4 TB/s × 1000) / 1000 = 4,000 GB/s per GPU (since 4 TB = 4,000 GB).
So each GPU gets 4,000 GB/s, which is exactly the per-GPU bandwidth. The total system bandwidth is 4 TB/s × 1000 = 4,000 TB/s.
This simple calculation shows that if you double the number of GPUs, the total bandwidth scales linearly, but the per-GPU bandwidth stays the same. The real issue is that in practice, adding GPUs introduces communication overhead that reduces the effective per-GPU bandwidth.
Background: Why Memory Bandwidth Matters
Modern AI models, especially large language models (LLMs) and transformers, are memory-bound. During training, the GPU must constantly read and write model weights, activations, and gradients. During inference, it must stream the entire model through the processor for every token generated. The faster the memory can feed data, the faster the computation completes. A GPU with more compute cores but insufficient memory bandwidth will sit idle waiting for data.
The classic example is the attention mechanism in transformers: it requires reading the entire key-value cache for each generated token. As the context length grows, the memory traffic grows quadratically. Even with efficient kernels, the memory system becomes the limiting factor.
How It Works: The Memory Wall
The “memory wall” refers to the growing gap between processor speed and memory speed. While transistor counts and compute throughput have followed Moore’s Law, memory bandwidth has increased at a slower pace. For GPUs, high-bandwidth memory (HBM) was developed to close this gap, but it is expensive and has physical limits.
When you add more GPUs to a data center, you also add more memory and more bandwidth, but you also add communication overhead. Data must be exchanged between GPUs via interconnects like NVLink or InfiniBand. If the interconnect bandwidth is lower than the memory bandwidth, the system becomes communication-bound. Even with perfect scaling, the per-GPU memory bandwidth does not increase; it stays constant. So adding GPUs only helps if the workload is compute-bound, which is rare for large models.
Common Problems: Bottlenecks and Diminishing Returns
One common misconception is that buying more GPUs always speeds up training. In reality, distributed training has overheads: synchronization, gradient averaging, and data shuffling. When the model is large, these overheads can dominate. The result is that doubling the GPU count might only yield a 1.5x speedup, or even less, due to memory bandwidth saturation.
Another problem is the cost. High-bandwidth memory (HBM) is expensive, and the packaging required to stack it near the GPU die is complex. The cost per GB of bandwidth is much higher than for traditional DDR memory. Data centers also face power and cooling constraints, so adding more GPUs increases operational costs without proportional performance gains.
How to Choose: Memory Bandwidth vs. Compute
When selecting GPUs for AI workloads, consider the ratio of memory bandwidth to compute (often measured in bytes per FLOP). For memory-bound models, you want a high ratio. For compute-bound models, you can tolerate a lower ratio. A rule of thumb is that for transformer-based models, you need at least 1 byte of memory bandwidth per FLOP of compute to avoid stalling.
Also consider the interconnect. If you plan to scale to hundreds of GPUs, the network between them must have sufficient bandwidth. NVLink and other high-speed interconnects are critical. The total system bandwidth is the sum of memory bandwidth across all GPUs, but the effective bandwidth is limited by the slowest link.
Comparisons: HBM vs. GDDR vs. DDR
GPUs for AI typically use HBM (High Bandwidth Memory) or GDDR (Graphics Double Data Rate). HBM offers much higher bandwidth (up to 4 TB/s per GPU) but is more expensive and has lower capacity. GDDR is cheaper and has higher capacity but lower bandwidth (around 1 TB/s). DDR is used in CPUs and has even lower bandwidth.
For data center AI, HBM is the standard because the memory bandwidth is the primary constraint. However, some newer approaches use near-memory computing or processing-in-memory (PIM) to reduce data movement. These are still experimental but show promise.
Costs and Alternatives
Increasing memory bandwidth is not just about buying better GPUs. It also involves designing the data center to minimize data movement. Techniques include:
- Model parallelism: splitting the model across GPUs to reduce the amount of data each GPU needs.
- Gradient compression: reducing the size of gradients exchanged between GPUs.
- Mixed precision training: using lower precision (e.g., FP16 or BF16) to reduce memory traffic.
- Memory pooling: sharing memory across GPUs via technologies like NVLink-C2C.
Another alternative is to use specialized hardware like Google’s TPU, which has a different memory architecture, or to use CPU-based systems with large memory bandwidth for certain workloads.
Frequently Asked Questions
Why can’t I just add more GPUs to solve the memory bandwidth problem?
Adding more GPUs increases total bandwidth, but the per-GPU bandwidth stays the same. The bottleneck is often the interconnect between GPUs, not the memory itself. If the model is already memory-bound on a single GPU, adding more GPUs will not help unless the model is parallelized in a way that reduces memory traffic per GPU.
What is the difference between memory bandwidth and memory capacity?
Memory bandwidth is the rate at which data can be read from or written to memory (GB/s or TB/s). Memory capacity is the total amount of data that can be stored (GB or TB). AI models need both: enough capacity to hold the model and enough bandwidth to feed the compute units quickly.
How do I know if my AI workload is memory-bound or compute-bound?
You can profile the workload using tools like NVIDIA’s Nsight or PyTorch’s profiler. If the GPU utilization is low but memory bandwidth is near 100%, it’s memory-bound. If compute utilization is high and memory bandwidth is low, it’s compute-bound. In practice, most LLM inference is memory-bound.
Is HBM always better than GDDR?
Not always. HBM offers higher bandwidth but lower capacity and higher cost. GDDR is cheaper and has higher capacity, making it suitable for graphics or workloads that need large memory but not extreme bandwidth. For AI training and inference, HBM is usually preferred because the bandwidth is the limiting factor.
What is the future of memory bandwidth in AI?
Emerging technologies like processing-in-memory (PIM), optical interconnects, and advanced packaging (like chiplets) aim to increase bandwidth while reducing energy. For example, Samsung’s HBM-PIM and other research prototypes show promise. However, these are not yet mainstream.
Related
Formula used
in t | Training time | hours | 100 | 0 in b | Memory bandwidth | TB/s | 4 | 0 in g | Number of GPUs | - | 1000 | 1 out eff | Effective bandwidth per GPU | GB/s | (b * 1000) / g | 2 out total_bw | Total system bandwidth | TB/s | b * g | 2
Each line is one input (in, sel, list) or one result (out: label, unit, formula, decimals). How the calculators work
Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.
More cards in this drawer
- Uninstall MATLAB Mac: Full Removal Guide
- Data Center Firewall: How It Works & How to Choose
- Best CPUs for Gaming: Top Picks & Buyer’s Guide
- Galaxy S22 Ultra: Specs, Features & Verdict
- Samsung Contacts Disappeared? Fix It Now
- Swim With Brainrot Codes? Meaning & Use