AI Data Center Network Fabrics: How GPUs Connect

AI data center network fabrics connect GPUs across servers so they can exchange training data, gradients, and other workload traffic quickly and predictably. This guide explains how to plan and bring up that network; it assumes you have GPU servers, compatible network adapters, switches, cabling, and an operating system or cluster stack ready to configure.

What You’ll Need

• GPU servers with network interface cards (NICs) that support the chosen fabric and workload communication libraries.
• Compatible switches and cables or optical transceivers, with enough ports and bandwidth for the planned links.
• A fabric design, including topology, addressing, routing, and a method for managing congestion.
• Host software and drivers for the NICs, plus the communication stack required by your applications, such as MPI or a GPU collective library.
• Administrative access to configure switches and hosts, and a way to run link, bandwidth, and application-level tests.

Step-by-Step

1. Define the workload and traffic pattern. Identify how many GPUs and servers must communicate, whether traffic is mostly within a server or across servers, and whether jobs use frequent all-reduce operations, parameter exchange, or distributed storage. Estimate peak data movement and acceptable latency from workload requirements. Do not size the network from the total GPU count alone: placement, concurrent jobs, and communication patterns matter.
2. Choose a fabric technology and end-to-end stack. Common options include Ethernet, often with RoCE (RDMA over Converged Ethernet), and InfiniBand. Both can support high-throughput, low-latency GPU communication when the NICs, switches, drivers, and software are configured compatibly. Check support across every component, including the GPU communication library. RDMA can reduce CPU involvement, but it does not remove the need to configure and monitor the network correctly.
3. Design the topology and capacity. Select a topology such as a leaf-spine fabric, then calculate the required server-facing ports and uplinks. Aim for sufficient path capacity between racks; oversubscription means the total host-facing bandwidth exceeds the bandwidth available on shared uplinks. That may be acceptable for some workloads, but can limit synchronized GPU jobs. Plan redundant paths where availability requirements justify them, and verify that redundancy does not create unexpected routing or congestion behavior.
4. Map GPUs and NICs to the fabric. Document which NIC ports connect to which switches, and how each server’s GPUs communicate through its PCIe or other host interconnect. On multi-NIC systems, confirm the operating system and application can use the intended interfaces. Keep a clear mapping of GPU, NIC, port, switch, and address. This makes it easier to detect a slow or misrouted path later.
5. Install and configure host networking. Install supported NIC drivers and firmware, configure interface names, addresses, MTU, and routing, and verify link state and negotiated speed. For RoCE, coordinate settings across hosts and switches, including traffic classification and congestion controls appropriate to the design. For InfiniBand, configure the required fabric management and partitioning. Avoid copying settings from a different environment without checking compatibility.
6. Configure switching and traffic behavior. Apply the intended VLANs or InfiniBand partitions, routes, link aggregation or multipathing where designed, and quality-of-service settings. For RoCE deployments, ensure the switch and NIC configuration consistently handles the selected priority and congestion-notification mechanisms. Lossless behavior is not a magic setting: poorly configured pause behavior can spread congestion. Follow the equipment and software documentation, and validate counters rather than assuming the configuration works.
7. Validate basic connectivity before GPU testing. Check that every expected link is up at the correct speed, hosts can reach their intended peers, and routing takes the designed paths. Inspect interface, switch, and NIC counters for errors, drops, pause events, or retransmissions. Test one server pair first, then expand to a rack and the full fabric. Resolve physical and addressing problems before interpreting application results.
8. Benchmark the fabric and application path. Use supported point-to-point bandwidth and latency tests, followed by collective communication tests that resemble the actual GPU workload. Test one pair, multiple pairs, and concurrent traffic. Confirm the application selects the intended NICs and that GPU-to-network data movement is supported by the installed stack. Compare results with the platform’s documented expectations, not an unrelated benchmark’s headline number.
9. Monitor under representative load. Run realistic jobs and watch link utilization, queue or congestion indicators, drops, NIC errors, and application communication time. Check whether slowdowns are isolated to a host, rack, route, or workload phase. Keep configuration records and establish baseline measurements so changes can be compared. Recheck performance after driver, firmware, switch, or application updates.

Common Mistakes

• Planning from link speed alone: A fast port does not guarantee fast end-to-end communication if uplinks are oversubscribed or traffic is concentrated on one path.
• Mixing incompatible settings: Different MTUs, traffic priorities, firmware, or driver capabilities can cause drops or prevent RDMA from working as intended.
• Assuming all NICs are used: The operating system or communication library may select only one interface unless routing, interface selection, or library configuration is correct.
• Ignoring congestion counters: A link can remain up while queues, retransmissions, or drops damage collective performance.
• Testing only one pair of servers: A simple test can pass even when the full fabric is congested under simultaneous GPU traffic.

Tips

• Keep a port-to-host map and record switch, NIC, firmware, driver, and software versions.
• Change one class of settings at a time, then repeat the same test to identify the effect.
• Separate management traffic from high-volume GPU data traffic when the design and available interfaces permit it.
• Use consistent host configuration and automate checks for addresses, MTU, link state, and selected interfaces.
• Measure application communication time as well as raw network throughput; the workload’s collective behavior is what ultimately matters.

Troubleshooting

No link or unexpected link speed: Check cable and transceiver compatibility, seating, port configuration, and NIC firmware. Try a known-good cable or port, then inspect both ends’ link and error counters.

Hosts cannot communicate: Verify addresses, subnet or partition membership, routes, VLAN configuration, and firewall rules. Test progressively from interface status to peer reachability, and confirm the intended interface is active.

RDMA tests fail while ordinary traffic works: Check that RDMA is enabled and supported end to end, then verify drivers, device selection, traffic priority, MTU, and congestion settings. Consult the specific stack’s diagnostics; normal IP connectivity alone does not prove RDMA is operational.

Performance is lower than expected: Check for a slower negotiated link, wrong NIC selection, oversubscribed uplinks, uneven routing, retransmissions, drops, or congestion. Compare single-pair tests with many-to-many tests to distinguish a host issue from a fabric bottleneck.

GPU jobs slow down only at scale: Inspect collective communication timing and network counters during the job. Test fewer nodes, then add nodes in stages. If performance falls at a particular scale, investigate shared uplinks, hot spots, and job placement before changing application settings.

Frequently Asked Questions

What is a network fabric in an AI data center?
A network fabric is the connected system of switches, links, NICs, and configuration that carries traffic among servers and their GPUs.

Do GPUs connect directly to each other over the fabric?
Usually, a GPU sends data through its server’s host interconnect and NIC, across the network, and to another server’s NIC and GPU. Some systems provide additional specialized paths, but the exact route depends on the hardware and software.

Is Ethernet suitable for connecting GPUs?
Yes. Ethernet can carry GPU traffic, including RDMA-based RoCE deployments, when components and settings are compatible and the network is designed for the workload.

What is oversubscription?
It is the ratio of total bandwidth entering a network section to the bandwidth available on its shared uplinks. Higher oversubscription can reduce performance when many servers transmit at once.

How do I know whether the fabric is the bottleneck?
Compare network benchmarks with application communication timing, and inspect utilization, congestion, drops, and retransmissions while the workload runs. Also check whether the GPUs or host interconnects are already saturated.

Related

• Gothic 1 Remake Max Level: How High Can You Go?
• How to Fix LLM Responses That Forget Earlier Context
• Future of Creative AI: Predictions and Key Questions