LLM Inference Storage: Key Selection Factors

Choosing storage for LLM inference comes down to balancing three things: model loading speed, concurrent request throughput, and cost per gigabyte. The right setup serves the model weights fast enough that storage never becomes the bottleneck, while keeping the hot working set on the fastest tier and the rest on cheaper capacity.

Why Storage Matters for LLM Inference

LLM inference is compute-bound once the model is in GPU memory, but the storage layer still controls how quickly you get there. A 70B-parameter model in FP16 needs about 140 GB of weights. Loading that from a slow disk can take minutes, and every time a new GPU pod starts or a model version is swapped, that delay is pure downtime. More importantly, storage directly affects how many concurrent requests you can serve: if the model does not fit in a single GPU, weights are sharded across multiple GPUs, and each shard must be loaded from storage into the right GPU before serving starts. In multi-tenant environments, you also need to load different models for different users, and the storage system must handle those bursts without becoming the choke point.

How LLM Inference Reads Data

The read pattern for LLM inference is mostly sequential, large-block reads of weight files, not the small random reads typical of databases. Model weights are stored as tensor files (often in Hugging Face safetensors format), and the inference engine loads them in chunks that match the model’s tensor shapes. This means high sequential bandwidth matters more than high IOPS. However, there is a second, more latency-sensitive pattern: tokenization and de-tokenization vocabularies, and any embedding tables, are read frequently and can be small. If those live on a slow tier, every request pays a latency penalty. The dominant cost, though, is the initial load of the weights, which is why the storage system’s sustained read throughput is the key metric.

Key Storage Metrics to Evaluate

• Sequential read bandwidth – measured in GB/s. This determines how fast a model loads. A 140 GB model at 2 GB/s takes 70 seconds; at 10 GB/s it takes 14 seconds.
• Latency to first byte – important for the small file reads (tokenizer, config) that happen on every request path.
• IOPS – less critical for weight loading, but matters if you store many small files or have many concurrent model loads.
• Concurrency – how many parallel readers can saturate the storage. If you spin up 10 GPU nodes at once, all reading the same model, the storage must handle 10x the bandwidth.
• Consistency and caching – whether the storage layer can cache hot weights in RAM or NVMe to avoid hitting the network on every load.

Storage Tiers: Local, Network, and Object

Three main tiers are used in practice, and most production systems combine them:

• Local NVMe SSDs on the GPU node – the fastest option, with bandwidth in the tens of GB/s. The model is copied to the node’s local disk before loading into GPU memory. This is typical when you have a persistent pool of GPU nodes and you can pre-stage the model.
• Network-attached storage (NAS) or parallel file systems – shared storage like NFS or GPFS that all nodes can read. Convenient for multi-node inference where the same model must be available on every GPU. Bandwidth depends on the network and the file server’s capacity; 10-25 GB/s is common with high-end systems.
• Object storage (S3, GCS, Azure Blob) – the cheapest and most scalable, but with higher latency and lower bandwidth per connection. Used for the cold tier: model archives, old versions, and datasets. For inference, object storage is usually combined with a caching layer that pulls the model into local or network storage before serving.

A common pattern is a three-tier approach: object storage holds the master copy, a network file system caches frequently used models, and local NVMe holds the currently serving model. The storage choice is really about where you place the model and how often you move it.

How to Choose: A Decision Framework

Start by measuring your actual load pattern. Ask these questions:

1. How many GPU nodes are serving? If it’s one node, local NVMe is almost always sufficient. If it’s a cluster, you need shared storage or a distribution mechanism.
2. How often do you swap models? If you serve many small models that change frequently, you need a fast caching layer. If you serve one large model for weeks, the load time is amortized and you can afford slower storage.
3. What is your target time-to-first-token? This includes the model load time. If you need to serve a request immediately after a node starts, storage must be fast enough to load the model within your SLA.
4. What is your budget? Local NVMe is the most expensive per GB, object storage the cheapest. The right answer is often a mix: keep the hot model on NVMe, the warm models on network storage, and everything else in object storage.

For a single-node deployment, a simple rule: if the model fits in the node’s RAM, you can load it into page cache once and subsequent reads are from memory. In that case, the storage speed only matters for the first load. For multi-node, the network storage’s aggregate bandwidth must exceed the sum of the GPU nodes’ load rates, otherwise you’ll see a thundering herd effect when nodes restart simultaneously.

Common Problems and Misconceptions

A frequent mistake is assuming that high IOPS is what you need. LLM weight files are large and read sequentially, so a high-IOPS but low-bandwidth storage system (like a typical HDD RAID) will perform terribly. Another misconception is that object storage is fast enough because it’s

Related

• Pixel Overlord Codes: What They Are & How to Use Them
• 720p to 1080p Conversion: What It Means
• How to Clear Samsung Mailbox Email, Voicemail & Storage