400G and 800G Network Upgrades: AI Infrastructure Planning

400G and 800G network upgrades are the backbone of modern AI infrastructure, and planning them requires balancing bandwidth, power, cost, and future scalability. This article explains what AI infrastructure teams need to plan for when moving to 400G or 800G Ethernet, covering key considerations, architecture choices, and common pitfalls.

Why AI Workloads Drive the Need for 400G and 800G

AI training and inference workloads are massively parallel and data-intensive. Large language models and deep learning frameworks distribute computation across thousands of GPUs or TPUs, and these accelerators need to exchange data constantly—gradients, activations, and model parameters. The network fabric that connects them becomes the bottleneck if it cannot keep up with the compute. While 100G and 200G links can suffice for smaller clusters, the scale of modern AI training—often involving tens of thousands of accelerators—demands higher per-port bandwidth to reduce training time and improve utilization.

For example, a single GPU can generate tens of Gbps of traffic during training. With 8 GPUs per node, a single server can easily saturate a 400G NIC. As models grow, the collective communication patterns (like all-reduce) create incast and bursty traffic, which higher-speed links handle more gracefully. 800G, while still emerging, offers the next step in density and performance, enabling more bandwidth per rack and simplifying cabling.

Key Differences Between 400G and 800G

400G Ethernet is mature and widely deployed, with multiple form factors (QSFP-DD, OSFP) and a range of optics (SR8, DR4, FR4, LR4). It uses 4 lanes of 100G (PAM4) or 8 lanes of 50G. 800G, on the other hand, is the next generation, typically using 8 lanes of 100G (PAM4) and is designed to be backward compatible with 400G and 200G via breakouts.

The most significant practical difference is the number of fibers and the reach. 800G DR8 uses 8 parallel single-mode fibers, while 400G DR4 uses 4. This means cable plants must be planned for higher fiber counts. Additionally, 800G modules consume more power (typically 12-15W for SR8) and generate more heat, requiring better cooling in switches and servers.

For AI infrastructure teams, the choice between 400G and 800G is not just about speed—it’s about the entire ecosystem: switch ASICs, NICs, optics, cables, and the ability to scale. 400G has a proven ecosystem, while 800G is still maturing, with some components like NICs just starting to ship in volume.

Architecture Options: Fat-Tree, Spine-Leaf, and Beyond

Most AI clusters use a non-blocking fat-tree topology, where each leaf switch connects to a set of servers (or GPUs) and uplinks to multiple spine switches. The ratio of uplink bandwidth to server bandwidth determines the blocking factor. For AI training, a fully non-blocking fabric (1:1 oversubscription) is often preferred to avoid performance degradation.

With 400G, a typical design might have leaf switches with 32 or 64 ports of 400G, connecting to 32 or 64 servers (each with a 400G NIC) and uplinking to spines via 400G or 800G links. If you use 800G uplinks, you can halve the number of spine switches or increase the leaf-to-spine ratio. 800G also enables higher radix switches, like 64-port 800G, which can reduce the number of tiers needed.

Another emerging architecture is the use of optical circuit switching (OCS) for certain traffic patterns, but this is still niche. For most AI infrastructure teams, the decision is between building a dedicated AI fabric (often using Ethernet or InfiniBand) and a converged network. 400G and 800G Ethernet are increasingly the choice because of their openness and the ability to use standard tools.

Choosing Optics: SR, DR, FR, and LR

Optics are a major cost and planning factor. The table below summarizes common types for 400G and 800G:

• SR8/SR4 (short reach): For distances up to 100m (OM4 fiber). Uses multimode fiber (MMF). Cheapest for intra-rack or adjacent-rack connections.
• DR4/DR8 (datacenter reach): For 500m over single-mode fiber (SMF). Common for leaf-to-spine links in a datacenter hall.
• FR4/FR8 (fiber reach): For 2km over SMF, using CWDM wavelengths. Used for longer inter-building links.
• LR4/LR8 (long reach): For 10km over SMF. Typically used for campus or metro connections.

For AI clusters, most links are short (within a rack or between adjacent racks), so SR8 and DR4/DR8 dominate. However, as clusters scale, you may need FR for longer spine-to-spine or inter-pod links. The choice affects fiber types and cable plant design. For example, if you plan to use DR4 (single-mode), you need SMF patch panels, not MMF.

Power, Cooling, and Physical Planning

High-speed networking consumes significant power. A 400G QSFP-DD SR8 module draws about 10-12W, while an 800G OSFP SR8 draws 12-15W. A switch with 64 ports of 800G can consume over 1000W just for optics, plus the switch ASIC power. This impacts rack power budgets and cooling requirements.

AI servers themselves are power-hungry—a single 8-GPU server can draw 5-10kW. Adding multiple 400G NICs (each 10-15W) and the associated switch ports means you must plan for higher power density per rack. Liquid cooling is often needed for the highest-density racks, but even air-cooled racks require careful airflow design to handle the heat from optics.

Physical space is also a concern. 800G OSFP modules are larger than QSFP-DD, and switch faceplates may have lower port density. Cabling is thicker and less flexible, so cable management becomes critical. You may need to use breakout cables (e.g., 800G to 2x400G) to connect to existing 400G devices, which adds complexity.

Cost Considerations and Budgeting

Cost is a major factor. 800G optics are initially more expensive per Gbps than 400G, but the price gap narrows as volumes increase. However, the total cost of ownership includes not just optics but also switches, NICs, cables, and the cost of downtime during migration.

For a typical AI cluster of 1000 GPUs, you might need 125 leaf switches (each with 8x400G uplinks) and 8 spine switches (with 64x400G ports). Moving to 800G uplinks could halve the number of spine ports, but the 800G optics might cost more. A detailed TCO analysis is essential. Also consider the cost of new cabling—single-mode fiber is more expensive than multimode, but it supports longer reaches and future speeds.

Migration Strategies and Common Pitfalls

Migrating from 100G/200G to 400G or 800G requires careful planning. A common approach is to upgrade the spine first, using 800G links between spines, then gradually move leaves to 400G or 800G as NICs and servers are refreshed. Another strategy is to build a separate AI fabric for training workloads, leaving general-purpose traffic on the existing network.

Common pitfalls include:

• Underestimating fiber infrastructure: You may need to pull new fiber, especially if moving from multimode to single-mode.
• Ignoring optical power budgets: Long links with many patch panels can have high insertion loss, requiring higher-power optics or amplification.
• Not testing interoperability: Mixing vendors for switches, optics, and NICs can cause issues. Always validate in a lab.
• Overlooking NIC and driver support: Ensure your server OS and drivers support 400G/800G NICs.
• Forgetting about congestion control: AI traffic is bursty; you need proper flow control (like Priority Flow Control) and congestion management (like DCQCN) to avoid packet loss.

Frequently Asked Questions

Is 800G worth it for AI training?

For very large clusters (thousands of GPUs), 800G can reduce training time and simplify the fabric by reducing the number of cables and switches. However, for smaller clusters, 400G may be sufficient and more cost-effective. Evaluate your workload’s bandwidth requirements and growth plan.

Can I use existing fiber for 800G?

It depends. 800G DR8 uses 8 single-mode fibers, so you need enough dark fiber. If you have existing single-mode fiber with 8 fibers per link, it may work. Multimode fiber can support 800G SR8 up to 100m, but only if it’s OM4/OM5. Check your cable plant.

What is the difference between QSFP-DD and OSFP?

QSFP-DD and OSFP are both form factors for 400G and 800G modules. QSFP-DD is backward compatible with QSFP28 (100G) and is more common in switch and NIC designs. OSFP is slightly larger and has better thermal performance, often used for 800G. They are not physically compatible, so choose based on your hardware.

How do I calculate the number of spine switches needed?

For a non-blocking fabric, the total uplink bandwidth from leaves must equal the total server bandwidth. For example, if you have 32 leaf switches each with 32x400G ports to servers, you need 32*32*400G = 409.6Tbps of uplink. If you use 800G spine ports, you need 409.6Tbps / 800G = 512 spine ports. Divide by the number of ports per spine switch (e.g., 64) to get 8 spine switches.

What are the power requirements for 800G switches?

A 64-port 800G switch can consume 2-3kW or more, including optics. This is higher than a 400G switch. Ensure your racks can provide adequate power and cooling. For high-density AI racks, liquid cooling may be necessary.

Should I use Ethernet or InfiniBand for AI?

InfiniBand has historically offered lower latency and better performance for tightly coupled HPC workloads. However, Ethernet with RoCE (RDMA over Converged Ethernet) is now widely used for AI, offering similar performance with more flexibility and interoperability. 400G/800G Ethernet is often preferred for its ecosystem and ease of management.

Related

• Install Cosmic Desktop Fedora: A Complete Guide
• Genmoji Problems: Why It Feels Off
• How to Stereo Pair Apple HomePod