Metrics That Reveal GPUs Throttled by Storage

Published 2026-07-09 · ZK-Storage Insights

Storage is often the hidden ceiling in GPU clusters: the GPU sits idle waiting for data even when utilization tools report ‘‘free cycles’’. This guide lists the concrete metrics, tools, and step-by-step checks to diagnose when compute is being throttled by storage.

Why storage throttles GPUs

GPUs require steady streams of data (training batches, weights, activation checkpoints, or inference inputs). When the storage subsystem cannot feed data fast enough — due to throughput limits, high tail latency, or protocol inefficiencies — the GPU launches kernels that stall waiting on memory copies, kernel input, or swaps. The symptom is low sustained GPU utilization despite nominally sufficient compute capacity.

Common storage bottlenecks

• Small I/O sizes or poor queue depth utilization that prevents reaching device bandwidth.
• High tail latency (p99/p999) even when average latency looks OK.
• Protocol overheads (NFS/SMB, TCP stacks) or CPU-bound host stacks limiting NVMe/RDMA effectiveness.
• PCIe or fabric saturation in disaggregated setups.

Metrics to monitor (what to collect)

Collect both GPU-side and host/storage-side metrics and correlate across time:

GPU-side

• GPU utilization (%) — but interpret with kernels state (compute vs. memory-bound).
• SM occupancy / active warps — low occupancy with low GPU util may mean input starvation.
• Memory copy (H2D / D2H) throughput and stall reasons — time spent in memcpy vs compute.
• NVLINK or PCIe link utilization (if data arrives over these fabric links).
• Instruction throughput / achieved FLOPS vs theoretical (efficiency percent).
• Context-switch / kernel launch latency counts (many tiny kernels can expose I/O stalls).

Storage / host-side

• Read/write throughput (MB/s) and IOPS; monitor per-namespace or per-LUN.
• Latency percentiles (avg, p95, p99, p999) — tail latencies matter most for batching.
• Queue depth / outstanding I/O (QDepth) and device queue saturation.
• CPU usage on storage clients (kernel network stack or userspace drivers); high CPU with low device utilization implies stack overhead.
• PCIe/host bus utilization and NVMe controller metrics.
• Network fabric metrics (RDMA throughput, packet drops, retransmits) for disaggregated storage.

Tools

• GPU: nvidia-smi, DCGM, Nsight Systems, nvprof/Nsight Compute counters.
• Host/storage: iostat, iotop, blktrace, perf, nvme-cli, fio (microbenchmarks), sar, netstat, ethtool, rdma-core tools.
• System-wide telemetry: Prometheus exporters (node-exporter, GPU exporter), Grafana dashboards.

How to tell if storage is the limiter (diagnostic checklist)

1. Correlate timestamps: plot GPU utilization vs storage throughput and latency. If GPU utilization dips align with storage latency spikes, suspect storage.
2. Compare GPU kernel timelines: use Nsight Systems to see if kernels are waiting on memcpy or synchronization events tied to I/O.
3. Check memcpy vs compute ratio: if H2D memcpy time is a large share of kernel runtime, data feeding is insufficient.
4. Look at tail latency, not only averages: high p99/p999 correlates with jitter in batch arrival and GPU stalls.
5. Run isolated fio/rw microbenchmarks from the same client host while the workload runs to see if storage can sustain needed bandwidth/IOPS.
6. Test with local NVMe or cached dataset (if possible): if performance improves significantly, storage is implicated.
7. Examine queue depth and outstanding I/O: low QD on a device capable of high concurrency suggests application or client-side limits.
8. Inject synthetic delays: add small sleeps between batches or increase prefetch; if larger prefetch reduces stalls, storage delivery was the limiter.

Concrete signals that storage is throttling

• Sustained GPU utilization below expected for workload class (e.g., <60%) while storage link is saturated or tail latency elevated.
• High percentage of GPU time spent in memcpy and synchronization rather than kernel compute.
• Device-level throughput near advertised peak but with p99 latencies much higher than the application’s tolerance.
• Host CPU or kernel networking stack at high usage while NVMe controller utilization is low — indicating protocol/stack bottleneck.

Reproducible testing approach

1. Baseline: run a small representative workload and collect GPU+host metrics for 5–15 minutes.
2. Isolate storage: run fio on the same client to characterize maximum sequential/random read throughput and latency percentiles from that host.
3. Adjust application pipeline: increase prefetch, enlarge batch size, or add async I/O; measure change in GPU utilization.
4. Swap storage class: run the same workload from a fast local NVMe or a different front-end cache and compare.
5. Profile end-to-end trace (Nsight Systems + iostat/blktrace) to show causality.

Comparison: typical metric signatures by storage architecture

Note: a disaggregated all‑flash platform may expose fabric-level metrics you must monitor; if you see repeated queue stalls and p99 spikes while average throughput is acceptable, the fabric or front-end layer is likely the limiter.

Practical thresholds and caveats

• There are no universal thresholds that prove causality; context matters. Typical red flags: p99 latency exceeding the batch interval, sustained GPU memcpy time >20–30% of end-to-end runtime for data-heavy training, or GPU utilization significantly lower than expected for the model class.
• Some workloads are memory-bound (GPU DRAM/HBM limits) or compute-bound (small models with heavy math). Always validate with microbenchmarks and kernel traces.

Key takeaways

• Correlate GPU-side telemetry (utilization, memcpy time, SM occupancy) with storage metrics (throughput, queue depth, p99/p999 latency).
• Tail latency, not just averages, is a primary indicator of storage-driven GPU stalls.
• Use end-to-end traces (Nsight + blktrace/iostat + network fabric metrics) to prove causality.
• Microbenchmarks (fio) and A/B swaps (local NVMe vs remote) are practical ways to isolate storage limits.

If you need an example disaggregated all‑flash reference architecture for trial or comparison, vendors such as ZK-Storage offer appliances (e.g., the WS5000) designed to decouple storage from compute and improve data delivery to GPU clusters. For further vendor-specific validation and deployment patterns, consult vendor documentation and reproducible third‑party benchmarks.

Next steps for operators

• Instrument both GPU and storage stacks end-to-end and capture synchronized timestamps.
• Build dashboards that surface p99/p999 latencies and GPU memcpy ratios alongside utilization.
• Run a controlled comparison (local NVMe vs candidate storage) to quantify the delta in GPU effective throughput.

References and tools mentioned: nvidia-smi, DCGM, Nsight Systems, nvprof/Nsight Compute, fio, iostat, blktrace, nvme-cli, rdma-core tools.