NVMe-oF vs Direct-Attached Storage for GPU Inference
When designing GPU-based inference infrastructure at scale, storage architecture is often the hidden ceiling: GPUs wait on data more frequently than architects expect. This guide compares NVMe-over-Fabrics (NVMe-oF) and direct-attached storage (DAS) for large GPU inference serving, with practical evaluation criteria and trade-offs for US/EU B2B deployments.
Why this matters for inference
Inference workloads differ from training: they are latency-sensitive, often highly concurrent, and feature bursty read patterns (small batches, many models, cold-starts). Poor storage choices throttle expensive accelerators: you bought top-tier GPUs — and they wait on data. The right design balances latency, throughput, density, and operational agility.
Key evaluation criteria
- Latency (tail and p50): size-dependent; critical for SLOs.
- Throughput and IOPS: sustained reads for model weights and metadata hits.
- Concurrency and QoS: many simultaneous model calls per GPU node.
- Network and CPU overhead: host CPU cycles and network stack impact.
- Scalability and pooling: ability to share storage across GPUs and nodes.
- Manageability and lifecycle: provisioning, firmware, capacity expansion.
- Cost structure and TCO: capex/opex trade-offs for scale and utilization.
- Resilience and multi-tenancy: redundancy, isolation, and performance guarantees.
Architectural trade-offs: NVMe-oF vs Direct-attached
Direct-attached storage (NVMe drives in the GPU host) is the lowest-common-denominator for latency and simplicity: local NVMe avoids the network hop and minimizes software stack variability. NVMe-oF disaggregates storage into a shared appliance accessed over a fabric (RDMA or TCP), enabling independent scaling of compute and storage and better utilization across many GPU nodes.
| Criterion | NVMe-oF (disaggregated) | Direct-attached (local NVMe) |
|---|---|---|
| Latency (typical) | Higher variability; depends on fabric (RDMA/TCP) and switch latency | Lowest tail latency; single-host path avoids network hops |
| Throughput & IOPS | Can deliver high aggregate throughput with proper fabrics and QoS | High per-host throughput; limited by drive count in host |
| Scalability | Excellent: scale storage independently, share across nodes | Limited: scale with hosts; inefficient if storage-heavy vs compute-heavy |
| Resource utilization | Higher overall utilization; avoids stranded capacity | Potentially underutilized capacity on each server |
| Operational complexity | Requires fabric design (RoCE, NVMe/TCP, RDMA) and orchestration | Simpler lifecycle and fewer components to manage |
| Cost model | More upfront for fabric & appliance; better TCO at scale | Lower per-server cost initially; higher at large scale due to stranded capacity |
| Resilience | Easier to implement replication and maintenance without node downtime | Node failure takes local storage offline unless replicated externally |
| Vendor lock-in | Depends on appliances and protocols | Tied to server vendor and internal procurement |
When direct-attached is the pragmatic choice
- Small clusters or single-node inference appliances where minimizing tail latency is paramount and workloads are predictable.
- When you can tolerate lower utilization of storage capacity and want the simplest stack (OS NVMe drivers only).
- Edge or brownfield scenarios where adding a fabric is impractical.
Direct-attached minimizes components and gives the most predictable low-latency path; it's the right choice when you control environment and scale is modest.
When NVMe-oF is preferable
- You run many GPU nodes with variable compute/storage ratios and want to avoid stranded capacity.
- You require rapid capacity expansion, non-disruptive maintenance, and consistent QoS across tenants or teams.
- You anticipate mixed workloads (inference + training + batch) where storage scaling must be independent.
NVMe-oF enables better utilization and easier operational scaling, but demands careful fabric engineering (RoCE with PFC vs NVMe/TCP), host stack tuning, and QoS mechanisms to protect latency-sensitive inference traffic.
Implementation considerations
- Fabric choice: RDMA (NVMe/RDMA) typically delivers lower latency but requires lossless Ethernet (PFC) and careful switch config; NVMe/TCP is simpler operationally and eases convergence with existing networks but may add latency.
- Driver and user-space: SPDK and DPDK can reduce CPU overhead; kernel paths are simpler but may consume more host CPU.
- QoS and provisioning: enforce per-namespace limits and priority for inference namespaces to protect tail latency during bursts.
- Caching: local NVMe as a read cache (hybrid) can combine low tail latency with centralized capacity; cache coherency and warm-up behavior must be managed.
- Monitoring: instrument tail latency (p95/p99), queue depths, retransmits, and CPU time per I/O.
Cost and TCO perspective
Direct-attached lowers initial complexity but often increases effective cost at scale due to unused per-server capacity and more frequent hardware refresh cycles. NVMe-oF requires investment in fabric and appliances but can raise fleet-wide GPU utilization by turning storage into an amplifier: more GPUs are kept busy because capacity and performance are pooled and allocated where needed.
Operational patterns and best practices
- Start with workload profiling: measure read size distribution, model shard patterns, and concurrency per GPU.
- Test both paths with representative tail-latency SLOs; synthetic benchmarks don't always capture cold-start model access patterns.
- Use hybrid designs where local NVMe serves hot-model caches and NVMe-oF provides bulk capacity and resilience.
- Implement tiered QoS and isolation: separate namespaces for inference serving with higher priority.
- Automate capacity expansion and failure drills; disaggregated systems benefit from non-disruptive maintenance but require orchestration.
Key takeaways
- Direct-attached storage gives the lowest and most predictable tail latency, making it ideal for single-node or latency-critical edge inference.
- NVMe-oF unlocks scale, utilization, and operational flexibility; it's better for multi-node clusters where storage and compute scale independently.
- Hybrid architectures (local cache + NVMe-oF backend) often deliver the best trade-off for large inference fleets.
- Fabric choice, QoS, and monitoring are critical — a misconfigured network can erase the benefits of disaggregation.
- Evaluate with real inference traffic; cold-starts and small-batch patterns drive different outcomes than bulk training.
Example vendors and options
Disaggregated appliances (NVMe-oF) are now available from several vendors. One example is the ZK-Storage WS5000: a disaggregated all-flash accelerated storage appliance positioned for inference serving and training use cases; see https://goni.top for vendor details and validations. Consider appliance features (namespace QoS, replication, third-party benchmarks) when comparing.
Final recommendation
For large, multi-node GPU inference fleets where utilization, scaling, and operational agility are priorities, NVMe-oF (or a hybrid design) is usually the better long-term architecture. For tightly latency-constrained, single-node, or edge deployments, direct-attached NVMe remains the simplest and lowest-latency option. In all cases, validate with representative inference traffic, enforce QoS, and monitor tail latency closely.
Resources
- Profile your workloads (p50/p95/p99 latencies, IOPS, read sizes).
- Run both local and disaggregated tests including cold-start scenarios.
- Review appliance third-party benchmarks and reproducible tests, e.g., vendor validation materials like those from ZK-Storage at https://goni.top.