NVMe over Fabrics vs Local NVMe for AI and HPC

NVMe over Fabrics (NVMe-oF) and local NVMe storage both deliver high-performance flash, but for AI and HPC workloads the choice comes down to whether you need a shared, scalable data pool or the lowest possible latency for a single node. Local NVMe is the fastest and simplest option for per-node scratch and model training, while NVMe-oF provides shared storage that lets many compute nodes access the same dataset without copying it, at a small latency cost. This article explains how each works, where they excel, and how to decide based on your workload’s access patterns, scale, and budget.

Formula

The calculator above computes a simple aggregate throughput metric: Effective throughput = IOPS × Bandwidth (GB/s). This is not a standard benchmark, but it gives a rough idea of the combined capability of a storage system when both random I/O (IOPS) and sequential bandwidth matter. For AI training, you often need both: high IOPS for small metadata operations and high bandwidth for streaming large tensor files. The variables are:

• IOPS: Input/output operations per second, a measure of random access speed.
• Latency (µs): Time for a single I/O operation to complete, in microseconds. (Not used in the formula but shown for context.)
• Bandwidth (GB/s): Sustained data transfer rate in gigabytes per second.

The result is a unitless number that helps compare systems when both metrics are relevant. A higher number indicates a system that can handle more simultaneous work.

Worked Example

Using the default values from the calculator: IOPS = 100,000 and Bandwidth = 10 GB/s.

Effective throughput = 100,000 × 10 = 1,000,000.

So the effective throughput is 1,000,000 (unitless). This is not a real-world benchmark but a comparative figure. For instance, a local NVMe drive might achieve 500,000 IOPS and 7 GB/s, giving 3,500,000, while an NVMe-oF fabric with 200,000 IOPS and 5 GB/s gives 1,000,000. The local drive appears better in this crude measure, but the fabric allows many nodes to share that performance, which the calculator doesn’t capture.

Background: Why Storage Matters for AI and HPC

AI training and HPC simulations are I/O-intensive. Model training reads massive datasets repeatedly, writes checkpoints, and logs metrics. HPC applications often produce large output files and need to read input data across thousands of cores. Storage performance directly impacts how long a training epoch takes or how fast a simulation can write results. Historically, HPC used parallel file systems (like Lustre or GPFS) over network storage, but NVMe has changed the game by offering sub-100-microsecond latency and millions of IOPS. Local NVMe provides the lowest latency because the drive is attached directly to the PCIe bus. NVMe-oF extends that performance over a network, enabling shared access.

How Local NVMe Works

Local NVMe drives are installed directly in a server, connected via PCIe lanes. The operating system sees them as block devices, and applications can access them with minimal overhead. For AI training, a common pattern is to use local NVMe as a fast scratch space: copy the dataset from a slower shared storage to local drives, train, then discard the copy. This avoids network latency during the most I/O-intensive phase. The downside is that local storage is not shared—each node has its own copy, which requires data staging and can lead to inconsistent states if not managed carefully. Also, if a node fails, its local data is lost unless replicated elsewhere.

How NVMe over Fabrics Works

NVMe-oF is a protocol that transports NVMe commands over a network fabric, typically Ethernet (using RDMA or TCP) or Fibre Channel. It presents remote NVMe drives as if they were local block devices. The key advantage is that multiple servers can access the same storage pool concurrently, which is essential for distributed training where each GPU reads from the same dataset. NVMe-oF supports both block-level (NVMe-oF) and file-level (NVMe over Fabrics with NVMf) access, but block-level is more common for low latency. The fabric adds a few microseconds of latency compared to local PCIe, but still far less than traditional network storage like iSCSI or NFS. Modern NVMe-oF implementations can achieve latencies around 100-200 microseconds, compared to 20-50 microseconds for local NVMe.

Comparing Performance: Latency, Bandwidth, and Scalability

When comparing the two, look at three metrics:

• Latency: Local NVMe typically has 20-50 µs latency; NVMe-oF adds 50-150 µs depending on the network. For AI workloads that require many small random reads (e.g., data augmentation), lower latency helps, but the difference is often negligible when using large batch sizes.
• Bandwidth: A single local NVMe drive can deliver 3-7 GB/s. A shared NVMe-oF system can aggregate dozens of drives to deliver hundreds of GB/s, but that bandwidth is shared among clients. For a single node, local wins; for a cluster, the fabric’s aggregate bandwidth is far higher.
• Scalability: Local NVMe scales with the number of nodes, but each node has a fixed capacity and performance. NVMe-oF scales independently—you can add storage nodes or expand capacity without touching compute nodes. This is crucial for AI training where datasets grow over time.

A common misconception is that NVMe-oF is always slower. In practice, for large sequential reads (like streaming a 100 GB dataset), the network fabric can saturate the link, and the bottleneck becomes the network, not the storage. With RDMA, the CPU overhead is low, and the bandwidth can match local PCIe for large transfers.

How to Choose: Workload Patterns and Architecture

The decision depends on how your application accesses data:

• Single-node training: If you train on one server with multiple GPUs, local NVMe is often sufficient and cheapest. You can store the dataset on a local drive and avoid network overhead.
• Distributed training: If you use multiple nodes (e.g., with Horovod or PyTorch DDP), shared NVMe-oF avoids copying data to every node. Each node reads from the same pool, ensuring consistency and simplifying data management.
• Checkpointing: AI training writes checkpoints frequently. With local NVMe, checkpoints are stored on each node, which is fine if you only need them for that node. But if you need to recover from a node failure, you need a shared location. NVMe-oF provides a natural checkpoint repository.
• HPC simulations: Many HPC applications use MPI and require shared files. NVMe-oF with a parallel file system (like BeeGFS or Lustre over NVMe) is the standard approach. Local NVMe is used as a burst buffer—a fast cache between compute and the shared storage.

A hybrid approach is common: use local NVMe for temporary data, and NVMe-oF for persistent shared data. For example, a training job might first copy the dataset from NVMe-oF to local NVMe, then train, then write the final model back to the shared pool.

Costs and Management Considerations

Local NVMe is cheaper upfront because you only pay for the drives and the server slots. However, you may need more total capacity because each node holds a copy of the dataset. NVMe-oF requires additional hardware: network adapters (RDMA-capable), switches, and a storage server with many NVMe drives. The total cost can be higher, but it reduces data duplication and simplifies management. Also, NVMe-oF systems often include features like snapshots, replication, and quality-of-service, which are harder to implement with local drives. For large clusters, the operational savings can outweigh the hardware cost.

Common Problems and Misconceptions

One common issue is that NVMe-oF over TCP (as opposed to RDMA) has higher CPU usage and latency. If your network doesn’t support RDMA, you might see performance close to iSCSI. Another issue is that not all applications are designed for shared storage—some expect a local file system and may not handle network file locking well. Also, people often assume that NVMe-oF is a file system, but it’s a block protocol. You still need a file system (like ext4 or XFS) on top, and if multiple nodes mount the same block device, you need a cluster file system like OCFS2 or GFS2, which adds complexity. A simpler approach is to use NVMe-oF as the backend for a distributed file system like BeeGFS or Lustre, which handles file locking and metadata.

Frequently Asked Questions

Is NVMe-oF as fast as local NVMe?

Not quite. NVMe-oF adds network latency, typically 50-150 microseconds, compared to 20-50 microseconds for local NVMe. For large sequential transfers, the difference is small, but for small random I/O, local is faster. However, NVMe-oF provides shared access, which is essential for cluster workloads.

Can I use NVMe-oF for AI training?

Yes, many AI clusters use NVMe-oF to share datasets across nodes. It works well for large datasets that don’t fit on a single node’s local drives. The key is to ensure your network supports RDMA for low latency and high bandwidth.

What is the difference between NVMe-oF and NVMe over TCP?

NVMe-oF is the general protocol; NVMe over TCP is one transport option. TCP is more compatible but has higher CPU overhead and latency than RDMA. RDMA (e.g., RoCE or InfiniBand) is preferred for performance-critical workloads.

Do I need a parallel file system with NVMe-oF?

If you only have one node accessing the storage, you can use a regular file system. For multiple nodes, you need a shared file system that supports locking, such as BeeGFS, Lustre, or a cluster file system like OCFS2. Otherwise, data corruption can occur.

How do I decide between local NVMe and NVMe-oF for a small cluster?

If you have fewer than 4 nodes and the dataset fits on each node’s local drives, local NVMe is simpler and cheaper. If the dataset is larger than a single drive or you need to share data without copying, NVMe-oF is worth the investment.

What is the typical latency of NVMe-oF over RDMA?

With RDMA, latency is typically around 100-200 microseconds, compared to 20-50 microseconds for local NVMe. This is still much faster than traditional network storage like iSCSI (500+ microseconds).

Related

• Convert FAT32 to NTFS Without Losing Data? Yes, Here’s How
• 3D Printed Armor Safety & Comfort Guide
• Common Samsung Galaxy S8 Problems & Fixes

in io | IOPS | - | 100000 | 0 | 10000000
in lat | Latency (µs) | - | 100 | 0 | 10000
in bw | Bandwidth (GB/s) | - | 10 | 0 | 100
out total | Effective throughput (IOPS×GB/s) | - | io * bw | 0