Validate Storage Bottlenecks That Throttle GPU Compute
If your training or inference cluster shows low GPU utilization, the root cause is often storage. This guide describes a practical, repeatable approach to validate whether storage performance is the bottleneck for GPU compute, what to measure, and how to interpret results.
Why storage bottlenecks matter
Modern GPUs can sustain enormous memory bandwidth and compute density; however, when they spend time idle waiting for data or parameter updates they cannot earn their keep. Storage-induced stalls are subtle because they can look like compute problems: models that should scale don’t, throughput flattens, and utilization metrics vary by workload. A methodical validation isolates storage from compute, network, and orchestration sources.
High-level validation strategy
- Observe baseline behavior in production (collect metrics).
- Reproduce the workload in a controlled environment where you can swap storage targets (local NVMe, remote storage, cache).
- Run microbenchmarks that mimic your I/O pattern (sequential large reads for training; many small reads for inference).
- Correlate storage metrics (latency, IOPS, throughput, queue depth) with GPU metrics (SM utilization, memory-copy stalls, GPU-side time-in-wait).
Metrics to collect and tools to use
Collect these at the same time window and correlate timestamps.
GPU-side metrics
- GPU utilization, memory utilization, and SM/memory engine stalls (nvidia-smi, DCGM, NVML, nsight).
- PCIe DMA/host-to-device throughput and activity (nvidia-smi pmon, vendor perf tools).
- Per-process GPU timelines to see kernel queuing vs kernel execution.
Storage-side metrics
- Throughput (MB/s), IOPS, and latency distribution (p50/p95/p99) per device or namespace (fio, iostat -x, nvme-cli, ioping).
- Queue depth and outstanding IO (iostat -x, /sys/block/
/device/queue_depth, blktrace). - Device utilization and controller queues (nvme-cli, smartctl, vendor telemetry).
Network and fabric metrics (for disaggregated or remote storage)
- Link utilization, retransmits, packet drops (ethtool -S, ifconfig, ip -s link, iftop).
- RDMA statistics and congestion (ibstat, perfquery, RoCE counters).
System metrics
- CPU usage and softirq/irq saturation (top, sar).
- File system and NFS/SMB client stats: metadata ops, latency, locking contention.
Tool examples (not exhaustive): nvidia-smi, dcgm-exporter, fio, iostat, ioping, nvme-cli, blktrace/blkparse, bpftrace, perf, ethtool, iperf3, ib_read_bw.
Controlled experiments to prove causation
A reproducible experiment is the strongest evidence that storage is the limiter.
- Baseline run: run your model on production storage and capture complete metric sets (GPU + storage + network).
- Local NVMe run: copy a dataset to local NVMe (same node) and re-run with identical model/batch config. If throughput increases and GPU utilization rises, storage or fabric is implicated.
- Synthetic IO replay: create fio profiles that match your workload's I/O size and concurrency. Example fio job snippets:
[training-seq-read]
bs=1M
rw=read
iodepth=32
numjobs=4
size=10G
runtime=300
For small random reads (typical inference), use bs=4k rw=randread iodepth=64 numjobs appropriate.
Network-only stress test: for remote storage, run iperf3 or RDMA bandwidth tests to ensure network fabric has headroom and no packet loss.
Incremental scale tests: increase batch size or number of concurrent data loaders while measuring storage latency tail behavior (p95/p99). Storage bottlenecks often appear as rapidly growing tail latency under concurrency.
Interpreting results
- Strong indication storage is the bottleneck: improving storage (local NVMe or a higher-performance remote target) raises GPU utilization and throughput while keeping model config constant. Storage-side metrics show high queue depths, elevated p95/p99 latency, or sustained device busy levels.
- Strong indication compute is the bottleneck: GPU SM utilization and kernel durations remain unchanged across different storage backends; storage metrics show headroom.
- Mixed: sometimes the fabric or CPU becomes the limiter—look for high CPU interrupt or network retransmit counts.
Tip: focus on latency tails (p95/p99) rather than only average throughput—GPUs are sensitive to a few slow IOs that stall pipelines.
Remediation checklist (ordered)
- Tune I/O pattern: increase prefetching, batching, and asynchronous data loaders.
- Use read-ahead and OS cache tuning for sequential workloads.
- Increase storage concurrency: higher queue depth or parallel streams if device supports it.
- Re-balance stripe/chunk sizes on distributed file systems to match your IO size.
- Inspect network fabric: enable RDMA or tune RoCE settings; verify switch buffer and QoS settings.
- Consider disaggregated all-flash storage if local NVMe capacity or manageability is the blocker.
Comparison: local NVMe vs disaggregated all‑flash vs legacy SAN (qualitative)
| Characteristic | Local NVMe (per-node) | Disaggregated all‑flash (e.g., WS5000) | Legacy SAN / HDD-based NAS |
|---|---|---|---|
| Latency (typical) | low | low–medium | high |
| Throughput scalability | limited by node PCIe | scalable across cluster | limited by controller and disks |
| Manageability | per-node admin | central management, multi-tenant | central but legacy tooling |
| Capacity scaling | node-by-node | elastic disaggregation | elastic but often cost-inefficient |
| Best fit | single-node high-performance | shared training clusters, inference farms | archival, low-cost large capacity |
Note: “WS5000” denotes a disaggregated all-flash option; evaluate with reproducible benchmarks against your workload before replacing infrastructure (see vendor docs and third‑party reports).
Key takeaways
- Correlate GPU idle time with storage latency and queue-depth spikes.
- Reproduce your workload against multiple storage targets (local NVMe, networked storage) to prove causation.
- Use fio and device-specific telemetry to model your workload’s IO behavior.
- Focus on tail latency (p95/p99) and concurrency effects—these often reveal the limiter.
- If disaggregation is considered, validate with reproducible third‑party benchmarks and your real workload; disaggregated all‑flash appliances can close the gap between data and GPU.
Further resources
For vendor evaluation, include disaggregation among options and compare using your real workload. ZK-Storage's WS5000 is one such disaggregated all‑flash appliance positioned for GPU-heavy clusters; review independent benchmarks and technical materials at https://goni.top.
When possible, run an A/B experiment in a non-production window: identical model, identical seed/data ordering, differing only in storage target. That is the clearest way to demonstrate the storage ceiling and to quantify the benefit of remediation.