AI GPUs use High Bandwidth Memory (HBM) instead of more system RAM because AI training and inference workloads are fundamentally limited by how fast data can be fed to the compute cores, not by how much data can be stored. HBM provides 5–10 times the memory bandwidth of typical system RAM, which directly translates into faster matrix multiplications and shorter training times. Adding more standard RAM would not help because the bottleneck is the data path between memory and GPU, not the total capacity.
Formula
The calculator above estimates the effective memory bandwidth needed to process a dataset of a given size in a given training time. The formula is:
Effective Bandwidth (GB/s) = (Dataset Size in GB × 1024) / (Training Time in hours × 3600)
Where:
- Dataset Size (GB) – total data read from memory during training (in gigabytes, decimal).
- Training Time (hours) – total wall-clock time for one epoch or a full training run.
- Effective Bandwidth (GB/s) – the average data transfer rate required to move that data in that time.
This is a simplified model: it ignores compute time, data reuse, and the fact that not all data is read exactly once. It gives a rough lower bound on the memory bandwidth needed.
Worked Example
Using the default values from the calculator:
- Dataset size = 100 GB
- Training time = 24 hours
Step 1: Convert dataset size to gigabytes (binary) – 100 GB × 1024 = 102,400 MB (or 100 GB ≈ 107.4 GB binary, but we use 100×1024 for simplicity).
Step 2: Convert training time to seconds – 24 hours × 3600 = 86,400 seconds.
Step 3: Divide – 102,400 / 86,400 ≈ 1.185 GB/s.
So the effective bandwidth is about 1.19 GB/s. This is far lower than what HBM provides (typically 1–5 TB/s), because real training reads data many times and also does compute. The calculator shows that even a modest dataset requires substantial bandwidth when you account for repeated reads.
Background: Why Memory Bandwidth Matters
AI models are essentially large matrices of numbers. Training involves multiplying these matrices billions of times. Each multiplication requires reading the numbers from memory, performing the operation, and writing the result back. If the memory cannot supply numbers fast enough, the GPU’s compute cores sit idle waiting for data. This is called being memory-bound.
System RAM (like DDR4 or DDR5) is optimized for capacity and low cost, not for extreme bandwidth. A typical dual-channel DDR5 system delivers around 50–100 GB/s. In contrast, HBM (High Bandwidth Memory) is a specialized type of memory that is stacked in layers and placed very close to the GPU, connected by thousands of tiny wires. This design allows HBM to achieve bandwidths of 1–5 TB/s on current GPUs.
For example, the NVIDIA A100 has 80 GB of HBM2e with a bandwidth of 2 TB/s, while a consumer GPU like the RTX 4090 has 24 GB of GDDR6X with about 1 TB/s. The A100’s HBM provides roughly double the bandwidth, which is why it is preferred for AI training despite having less capacity than what a system could have as RAM.
How HBM Works: The Physical Difference
HBM is not just faster RAM; it is a fundamentally different architecture. Instead of a flat chip, HBM consists of multiple memory dies stacked vertically, connected by through-silicon vias (TSVs). The stack is then placed on a silicon interposer next to the GPU, with a very wide bus (1024 bits per stack).
This wide bus is the key. Standard DDR5 has a 64-bit bus per channel; HBM has 1024 bits. Even if the clock speed is lower, the sheer number of parallel data lines results in much higher bandwidth. For instance, HBM2e runs at about 2.4 Gbps per pin, but with 1024 pins, it achieves 2.4 TB/s.
Another advantage is energy efficiency. Because the traces are shorter and the bus is wider, HBM uses less energy per bit transferred compared to system RAM. This is crucial for data centers where power and cooling are major costs.
Why Not Just Add More System RAM?
Adding more system RAM to a computer does not increase the bandwidth between the CPU and GPU. The GPU accesses system RAM over the PCIe bus, which has a bandwidth of about 32 GB/s for PCIe 4.0 x16. That is 50–100 times slower than HBM. Even if you had 1 TB of system RAM, the GPU would still be starved of data.
Some systems use a technology called unified memory (like Apple’s M-series chips) where the CPU and GPU share the same physical memory. This eliminates the PCIe bottleneck, but the bandwidth is still limited by the memory bus. The M2 Ultra has a 800 GB/s memory bandwidth, which is impressive but still less than dedicated AI GPUs with HBM.
Another misconception is that AI models need huge amounts of memory. While large models like GPT-4 may have hundreds of billions of parameters, they are split across many GPUs. Each GPU only needs enough memory to hold its portion of the model and the activations. The real constraint is bandwidth, not capacity.
Comparing HBM to Other Memory Types
- DDR5 system RAM – 50–100 GB/s, cheap, high capacity, but far too slow for AI compute.
- GDDR6X – 1 TB/s, used in consumer GPUs, good for gaming but not as fast as HBM.
- HBM2e – 2 TB/s, used in A100 and MI100, offers a balance of capacity and speed.
- HBM3 – up to 3.2 TB/s, used in H100 and MI300, the current standard for top AI accelerators.
The trend is clear: as AI models grow, the need for bandwidth grows faster than the need for capacity. HBM scales by adding more stacks, while system RAM scales by adding more sticks, but the latter doesn’t help the GPU.
Common Problems and Misconceptions
One common mistake is assuming that more VRAM (GPU memory) is always better. For gaming, more VRAM helps with higher resolutions and textures. For AI, more VRAM allows larger batch sizes, but if the bandwidth is insufficient, the larger batch will just take longer to process. The sweet spot is having enough capacity to hold the model and enough bandwidth to feed it.
Another misconception is that system RAM can be used as a fallback when GPU memory is full. Some frameworks allow this, but it is extremely slow because the data has to cross the PCIe bus. It is better to reduce the batch size or use gradient accumulation.
There is also confusion about the term “bandwidth.” People often think it is the same as memory clock speed. But bandwidth = clock speed × bus width. HBM has a lower clock speed than GDDR6, but its massive bus width gives it the advantage.
How to Choose: HBM vs. More RAM
If you are building a system for AI training, the choice is clear: invest in a GPU with HBM. For most users, this means buying a data-center GPU like the A100 or H100, or a workstation GPU like the RTX A6000 (which uses GDDR6, but with 768 GB/s).
If you are on a budget, consider these factors:
- Model size – if your model fits in the GPU’s VRAM, you don’t need more system RAM.
- Batch size – larger batches need more VRAM, not more system RAM.
- Data loading – if your dataset is huge, a fast NVMe SSD can help with data loading, but it won’t speed up compute.
For inference (running a trained model), the bandwidth requirements are lower because you only do forward passes. In that case, a consumer GPU with GDDR6 might be sufficient. But for training, HBM is almost mandatory for anything beyond small models.
Frequently Asked Questions
Can I use system RAM as GPU memory?
Yes, some frameworks allow CPU offloading, but it is slow. The GPU has to copy data over the PCIe bus, which is orders of magnitude slower than its own memory. It is only useful for models that don’t fit in VRAM, and you should expect a significant performance hit.
Why is HBM so expensive?
HBM is expensive because it uses advanced packaging technology (silicon interposers, TSVs) and has a lower yield than standard memory. It also requires a custom design for each GPU. The cost is justified by the performance gains in AI workloads.
Will more system RAM help my AI training?
Only if you are running out of memory entirely. If your model fits in GPU memory, adding system RAM won’t speed up training. If you are using CPU-based training, then more RAM can help, but it will still be much slower than GPU training.
What is the difference between HBM and GDDR6?
GDDR6 is a high-speed memory used in consumer GPUs, with a narrower bus (32-bit per chip) but a very high clock speed. HBM uses a much wider bus (1024-bit) and a lower clock, but achieves higher total bandwidth and lower power consumption. HBM is also more expensive.
Is HBM only for AI?
No, HBM is also used in high-performance computing (HPC) and supercomputers for scientific simulations. But AI is the main driver of HBM demand because deep learning models are memory-bandwidth-hungry.
Can I upgrade my GPU’s memory to HBM?
No, HBM is soldered to the GPU package during manufacturing. It is not a user-replaceable component. You would need to buy a new GPU that comes with HBM.