Nvidia Vera Rubin NVL72 vs AMD Helios AI Rack

Nvidia’s Vera Rubin NVL72 and AMD’s Helios are next-generation AI rack-scale systems that take different approaches to scaling GPU compute, memory, and networking. The NVL72 is a liquid-cooled, 72-GPU rack with a custom NVLink domain, while AMD Helios is a rack-level architecture built around its Instinct accelerators and Infinity Fabric. This article compares their architectures, power, cooling, and practical deployment considerations.

Background: Why Rack-Scale AI Systems Matter

Training large language models and other AI workloads requires thousands of accelerators working in parallel. Traditional servers with 8 GPUs per node force heavy reliance on network switches to move data between nodes, which adds latency and power. Rack-scale systems integrate many GPUs into a single high-bandwidth domain, reducing the need for external networking and improving efficiency. Both Nvidia and AMD are moving toward this model, but they implement it differently.

Nvidia Vera Rubin NVL72 Architecture

The NVL72 is a rack-scale system that combines 72 of Nvidia’s next-generation Rubin GPUs with 36 Vera CPUs in a single liquid-cooled rack. The key innovation is the NVLink domain: all 72 GPUs are connected in a full mesh using NVLink 6, providing up to 130 TB/s of bisection bandwidth. This means every GPU can talk to every other GPU at high speed without going through a network switch. The system is designed to be a single, massive GPU, effectively, for the software stack.

• Compute: 72 Rubin GPUs, each with next-gen CUDA cores and tensor cores.
• Memory: Each GPU has HBM4 memory, with the entire rack pooling over 1 TB of high-bandwidth memory.
• Networking: NVLink 6 provides 130 TB/s of bandwidth within the rack, while external connectivity uses 1.6 TB/s Ethernet or InfiniBand.
• Cooling: Liquid cooling is mandatory; the rack is designed to dissipate up to 120 kW of heat.

From a software perspective, the NVL72 appears as a single CUDA device, which simplifies programming and reduces the overhead of data movement between nodes. This is a significant advantage for large-scale training jobs that would otherwise need to be split across multiple servers.

AMD Helios Architecture

AMD’s Helios is a rack-scale system built around its Instinct MI400 series accelerators (the successor to MI300). While AMD has not released full details, the architecture is expected to use a similar rack-scale approach but with Infinity Fabric as the interconnect. Helios is designed to be more modular and open, supporting industry-standard OCP (Open Compute Project) racks and allowing mixing of different accelerators.

• Compute: 32 to 64 Instinct MI400 GPUs per rack, depending on configuration.
• Memory: Each GPU has HBM3E or HBM4, with a similar pooled memory approach.
• Networking: Infinity Fabric provides high-bandwidth, low-latency connections between GPUs, but AMD also supports standard Ethernet and InfiniBand for scale-out.
• Cooling: Air cooling is possible for lower-power configurations, but liquid cooling is recommended for full performance.

AMD’s approach emphasizes compatibility with existing data center infrastructure. Helios racks can be deployed in standard 19-inch racks, and the software stack is based on ROCm, which is open-source and supports a wider range of frameworks.

Comparing the Two Architectures

The fundamental difference is the interconnect and the software ecosystem. Nvidia’s NVLink is a proprietary, high-bandwidth interconnect that has been optimized for years, and CUDA is the de facto standard for AI development. AMD’s Infinity Fabric is also high-bandwidth, but the ecosystem is smaller, and ROCm has historically had compatibility issues.

In terms of raw performance, Nvidia claims the NVL72 can train a 1-trillion-parameter model in 3 days, while AMD has not made comparable public claims. However, AMD’s open approach may appeal to organizations that want to avoid vendor lock-in or that already use open-source software.

Power and cooling are also differentiators. The NVL72 requires liquid cooling and is designed for high-density data centers. Helios offers more flexibility, with air-cooled options for smaller deployments, but at the cost of performance.

How to Choose Between NVL72 and Helios

Choosing between the two depends on your existing infrastructure, software stack, and performance requirements.

1. Software ecosystem: If your team uses CUDA and cuDNN, Nvidia is the safer choice. If you prefer open-source ROCm or need to support non-Nvidia accelerators, AMD is better.
2. Data center readiness: The NVL72 requires liquid cooling and high power capacity (up to 120 kW per rack). If your facility cannot support that, Helios with air cooling might be the only option.
3. Scalability: Both support multi-rack scaling, but Nvidia’s NVLink domain is more tightly integrated, which can reduce training time for very large models.
4. Cost: Nvidia systems typically have a higher upfront cost, but they also have a mature ecosystem that may reduce development time. AMD may offer lower hardware costs, but you may spend more on engineering to get comparable performance.

Common Problems and Misconceptions

A common misconception is that the NVL72 is simply 72 GPUs in a rack. In reality, it is a single logical GPU from the software perspective, which is both a strength and a limitation. If one GPU fails, the entire rack may need to be taken offline, whereas a traditional 8-GPU server allows for more granular fault tolerance.

Another issue is power density. Many data centers are not designed for 120 kW per rack, and retrofitting can be expensive. AMD’s Helios, with its air-cooled option, may be easier to deploy in existing facilities, but it may not deliver the same performance.

Finally, there is the question of software maturity. While ROCm has improved significantly, it still lags behind CUDA in terms of library support and community resources. This can lead to longer development cycles for AMD-based systems.

Costs and Total Cost of Ownership

Exact pricing is not public, but the total cost of ownership (TCO) includes hardware, power, cooling, and software licenses. Nvidia’s hardware is generally more expensive, but CUDA’s efficiency can reduce the number of GPUs needed for a given workload. AMD’s hardware may be cheaper, but if you need to hire engineers to port code from CUDA to ROCm, those costs can add up.

Power and cooling are significant ongoing costs. The NVL72’s 120 kW per rack means that a 10-rack deployment would consume 1.2 MW, which is substantial. Liquid cooling adds upfront costs but can reduce cooling energy compared to air conditioning. AMD’s air-cooled option saves on upfront costs but may increase energy bills over time.

Frequently Asked Questions

Can I use the NVL72 with existing CUDA code?

Yes, the NVL72 is designed to be fully compatible with CUDA. From a developer’s perspective, it appears as a single GPU with 72 times the memory and compute. Existing CUDA code that is written for a single GPU can run without modification, but you may need to adjust memory allocation to take advantage of the larger memory pool.

Does AMD Helios support PyTorch and TensorFlow?

Yes, both PyTorch and TensorFlow have ROCm versions that can run on AMD Instinct GPUs. However, some features may be missing or slower compared to CUDA. It is recommended to test your specific workload before committing to AMD hardware.

What is the difference between NVLink and Infinity Fabric?

NVLink is Nvidia’s proprietary interconnect that provides extremely high bandwidth (up to 130 TB/s in the NVL72). Infinity Fabric is AMD’s interconnect, which also provides high bandwidth but is designed to be more open and compatible with standard networking protocols. In practice, NVLink has lower latency and higher bandwidth, but Infinity Fabric is more flexible.

Can I mix Nvidia and AMD GPUs in the same rack?

No, the NVL72 and Helios are proprietary rack designs that only support their respective GPUs. However, at the data center level, you can have separate racks for each vendor. There is no standard for mixing them within a single rack-scale domain.

What are the power requirements for these systems?

The NVL72 requires up to 120 kW per rack, which typically means 480V AC or 600V DC power. Helios can be configured for lower power, around 60-80 kW, depending on the number of GPUs. Both require dedicated power infrastructure and, for full performance, liquid cooling.

Which system is better for inference as opposed to training?

For inference, the NVL72’s massive memory bandwidth can be overkill, but it allows for very large batch sizes and low latency. Helios may be more cost-effective for inference if you do not need the full 72-GPU domain. However, the choice depends on your specific inference workload and throughput requirements.

Related

• Hide Messenger Stories: How to Hide Everyone’s Stories
• Notion Plus Subscription: Does It Pay for Itself?
• ROM and RAM: Key Differences Explained