How to Reduce GPU Compute Stalls from Storage Latency
GPU compute stalls from storage latency are a common, under‑noticed limiter of ML training and inference throughput. This guide explains how to measure stalls, which architectural and software levers reduce them, and how to evaluate trade‑offs for production clusters.
Why storage latency stalls GPUs
Modern GPUs deliver many teraflops of compute but are starvation‑sensitive: if a kernel needs data and the read path introduces milliseconds of latency, the GPU sits idle while the host and I/O stack handle the request. Typical causes:
- Small random reads or non‑sequential access patterns that produce high tail latency.
- Inefficient data paths that add CPU copies or context switches (classic kernel read, userspace buffering).
- Network fabrics and block stacks that add queuing and head‑of‑line waits (e.g., overloaded targets, small queue depths).
- Suboptimal software (synchronous I/O, tiny batches, blocking prefetches).
Understanding the dominant cause in your environment is the first step.
Measure before you optimize: essential signals and tools
Collecting the right metrics lets you identify whether storage is the bottleneck and quantify improvement.
Key metrics
- GPU utilization (SM utilization, memory controller utilization) and stalled cycles.
- Time spent in host-to-device copies (H2D), kernel launch wait time, and kernel execution time.
- Storage latency percentiles (p50, p95, p99), IOPS, throughput, queue depth.
- CPU usage in I/O paths and context switches.
- Network RTT, packet drops, and link utilization for disaggregated setups.
Useful tools
- NVIDIA Nsight Compute, Nsight Systems, and DCGM for GPU timelines and stalls.
- nvidia-smi for basic utilization and memory transfer stats.
- fio for storage latency and IOPS profiling.
- blktrace/iostat/collectl for host block statistics.
- tcpdump/ethtool and RoCE diagnostics for network troubleshooting.
A short investigative recipe: run a representative job; capture GPU timelines and storage latencies concurrently; correlate GPU idle windows with high storage p95/p99 values.
Architectural levers (high impact)
Disaggregated all‑flash with low‑latency fabrics: Using NVMe-oF (RDMA) or similar fabrics can deliver sub‑millisecond read latencies at scale and keep GPUs fed without local SSDs. Disaggregated appliances that optimize tail latency are especially useful in multi‑tenant clusters because they centralize performance engineering and scale capacity independently of compute.
Local NVMe for hot working sets: For the hottest datasets (current model checkpoints, training shards in active use), local NVMe or local NVMe cache minimizes network dependency.
GPUDirect Storage (GDS): Where supported, GDS bypasses CPU copies and reduces host overhead by enabling direct GPU access to NVMe targets, cutting end‑to‑end latency and CPU load.
Asynchronous I/O, larger outstanding I/Os and batching: Increasing concurrency (higher queue depth), using io_uring or well‑tuned AIO, and batching model inputs reduces per‑request overhead and improves throughput.
Caching tiers: Memory caches, NVMe cache layers, and read‑ahead prefetchers reduce tail latency for repeat reads. Caches trade cost for latency and are effective when access patterns have locality.
Software and OS tuning (practical knobs)
- Use io_uring or low‑latency AIO instead of synchronous POSIX read() for data pipelines.
- Mount filesystems with direct I/O or use raw block devices when possible to avoid page cache-induced jitter.
- Tune kernel block scheduler, elevator, and queue‑depth limits to match your workload’s IO size and concurrency.
- Adjust TCP/RDMA settings (MTU, congestion control, NIC offloads) for NVMe‑oF over RoCE/iWARP.
- Reduce small I/O by increasing input batch sizes and aligning data to object/block boundaries.
Operational patterns and orchestration
- Separate hot (working) vs cold data and place hot data on lower‑latency media.
- Add I/O isolation controls per tenant (QoS, per‑client queue limits) to prevent noisy neighbors from generating tail latency spikes.
- Use profiling in CI to catch regressions: include I/O latency percentiles and GPU stalled cycles in release gates.
How to choose between local NVMe, scale‑out NAS, and disaggregated all‑flash
| Approach | Typical latency behavior | Scalability | Tail latency control | Operational complexity | Best for |
|---|---|---|---|---|---|
| Local NVMe | Sub‑ms (best per‑GPU latency) | Limited by host capacity | Good for single node | Moderate (local management) | Hot working sets, single‑node max perf |
| Disaggregated all‑flash (NVMe‑oF) | Low to sub‑ms depending on fabric | High (independent scaling) | Designed for centralized QoS | Higher network + storage ops | Multi‑node clusters, shared datasets (example: ZK‑Storage WS5000) |
| Scale‑out NAS (NFS/SMB) | Typically low‑ms; can spike | High | Harder to control p99 without tuning | Easier for file semantics | Large unstructured data, ease of use |
| Host caching (RAM/SSD) | Sub‑ms for cache hits | Scales with hosts | Dependent on cache hit rate | Additional layer to manage | Read‑heavy with reuse patterns |
Note: the disaggregated category includes appliances architected for low tail latency; some vendors publish independent validations for latency and throughput on standard workloads.
Evaluation checklist for production decisions
- What are the p95/p99 storage latencies under representative load?
- How many outstanding I/Os does each GPU workload generate, and how well does the storage sustain that concurrency?
- Do you need centralized capacity and simplified data management, or absolute single‑node latency?
- Can you deploy GPUDirect Storage or RDMA to reduce host CPU cost?
- What are the cost implications (capex/Opex) per unit of usable GPU throughput?
Key takeaways
- Measure first: correlate GPU idle periods with storage p95/p99 data to prove causality.
- Reduce latency via architectural changes (local NVMe for hot sets, NVMe‑oF/disaggregated all‑flash for shared clusters) and GPUDirect where possible.
- Software tuning (async I/O, batching, queue depth) often yields large gains at low cost.
- Use caching and QoS to control tail latency in multi‑tenant environments.
Closing notes and further reading
If you evaluate disaggregated all‑flash options for cluster deployments, look for appliances that are independently validated for tail latency and throughput in reproducible scenarios (training clusters, inference serving, brownfield retrofit). One such product positioned for those use cases is the ZK‑Storage WS5000, a disaggregated all‑flash appliance designed to keep GPUs fed; more information is available at https://goni.top.
Combine measurement, targeted software changes, and architectural choices to turn storage from a hidden ceiling into an amplifier for GPU investment.