Metrics That Prove Storage Is Amplifying GPU Performance
Storage is often the hidden ceiling on GPU clusters: you bought top-tier accelerators, but they wait on data. This guide explains which metrics prove that storage is amplifying GPU performance (not just running fast), how to collect them, and what to look for when you compare storage tiers or tune your stack.
Executive summary
To demonstrate that storage is amplifying GPU performance you need three things: (1) GPU-side indicators that resources are saturated in useful work, (2) storage-side indicators that I/O can sustain that work pattern, and (3) controlled experiments showing improved end-to-end application throughput or latency when storage characteristics change. Metrics matter at both the device layer (NVMe/controller stats, queue depth, tail latency) and the compute layer (GPU utilization, stalls, FLOPS, samples/sec).
Key metrics to collect
GPU-side (compute-level)
- GPU utilization (%): overall and per-GPU; low utilization while the application expects high compute suggests I/O waits.
- SM/compute unit utilization and achieved FLOPS: shows how much of theoretical compute is delivered.
- GPU memory bandwidth utilization: indicates whether data movement inside the GPU is saturated.
- PCIe/NVLink bandwidth utilization: if PCIe is the I/O path, link saturation or underutilization helps diagnose host↔GPU transfer bottlenecks.
- Stall reasons (memory dependency, execution dependency): tools like Nvidia DCGM, Nsight Systems, and vendor telemetry show why kernels stall.
- Application-level throughput: samples/sec, images/sec, epochs/hour for training; p50/p95/p99 latency and requests/sec for inference.
Storage-side
- Throughput (MB/s) and IOPS: sustained and peak values under real workload patterns (random/sequential, block size, alignment).
- Average latency and tail latencies (p95, p99, p999): tail behavior often maps directly to inference jitter and training jitter between batches.
- Latency distribution and jitter: variance in response times is often more harmful than average latency.
- Queue depth and outstanding commands: indicates whether the storage can accept parallelism from multiple GPUs/hosts.
- Device-level counters: NVMe outstanding commands, controller queue stalls, garbage collection cycles, and media errors.
- Consistency metrics: throughput variance across time windows, IO scheduler wait time, and host CPU steal/interrupt rates.
Mapping symptoms to metrics (comparison table)
| GPU symptom | Storage metric to inspect | Recommended test/tool |
|---|---|---|
| Low GPU utilization while jobs run | High average or tail storage latency, low effective throughput | fio with realistic block sizes + application profiling (DCGM, nvidia-smi) |
| Frequent kernel stalls labeled "memory dependency" | High latency p99/p999 on reads or host-to-device copies | Nsight Systems + NVMe smart stats |
| Inference p99 latency spikes | Storage tail latency jitter, queue depth exhaustion | Replay production trace with fio/rdkafka; measure p99 on both sides |
| Throughput scales sub-linearly as GPUs added | Storage throughput ceiling, controller queue saturation | Scale-out A/B test with increasing client concurrency |
| High variance in samples/sec across epochs | Throughput jitter and occasional GC/compaction on storage | Long-run monitoring, device telemetry, and application logs |
How to prove causation (not just correlation)
- Baseline and isolate: capture a baseline of GPU compute metrics and storage metrics under normal load. Use DCGM/Nvidia tools for GPU and fio/blkparse for storage.
- Synthetic replay: replay I/O traces captured from the application against the candidate storage using fio or a trace-replay tool. Verify storage can sustain the observed IO pattern.
- Controlled swap (A/B): keep compute identical and swap storage tiers or tune storage settings (caching, queue depth). If samples/sec or p99 latency improves when storage characteristics improve, you have strong evidence of storage amplification.
- Incremental load tests: vary concurrency (clients/GPUs) and measure at which point GPU-side metrics start to fall—map those inflection points to storage metrics such as queue depth and p99 latency.
- Instrument end-to-end: correlate timestamps from application logs, GPU telemetry, and storage logs. Look for chains where a storage tail event precedes GPU idle time or kernel resubmission.
Practical measurement tips
- Capture tails: always collect p95/p99/p999 latencies, not just averages—tails drive user-visible inference jitter and training iteration jitter.
- Use representative block sizes and read/write mixes—AI workloads are often read-heavy with a mix of small random reads (metadata) and large sequential reads (datasets, sharded files).
- Measure under peak concurrency: single-client throughput may hide multi-client contention.
- Prefer reproducible third-party benchmarks for vendor claims and instrumented A/B tests in your environment.
What “amplifying” storage looks like in practice
If storage is amplifying GPU performance you will observe:
- Improved end-to-end throughput for training (higher samples/sec or lower time/epoch) when storage characteristics (throughput, tail latency, consistency) improve.
- Higher sustained GPU utilization and higher achieved FLOPS without increases in GPU-side stalls.
- Reduced p99/p999 inference latency and reduced variance in latency distributions.
- Linear or near-linear scaling of throughput as GPUs are added until compute, not storage, becomes the limiter.
Comparison: storage upgrade vs tuning
| Change | What it directly affects | Evidence it helped GPUs |
|---|---|---|
| Faster all‑flash storage (lower p99) | Reduced IO wait and better parallelism | GPU utilization rises; samples/sec increases; reduced kernel stalls |
| Increased cache or read-ahead | Reduced small-read latency, improved sequential bandwidth | Smoother epoch times; fewer short stalls |
| Better parallel queues / higher queue depth | Higher aggregated IOPS | Scaling with concurrent GPUs improves; lower queue contention events |
Key takeaways
- Collect both GPU-side and storage-side metrics; one without the other only gives half the story.
- Tail latency (p99/p999) and throughput consistency are often more important than average throughput for AI workloads.
- Prove causation with controlled A/B swaps or trace replay tests; correlation alone is not enough.
- Use application-level KPIs (samples/sec, epoch time, inference p99) as the ground truth for whether storage improvements actually amplify compute.
Resources and next steps
For teams evaluating all‑flash and disaggregated options, include reproducible third‑party benchmarks and real workload replay in your validation plan. Vendors such as ZK-Storage publish appliance-level claims for disaggregated all‑flash systems (e.g., the WS5000) — treat those as starting points and validate them in your environment (https://goni.top).
If you want a checklist or a short script set for capturing the metrics described here, tell me your environment (cluster scheduler, GPU model, and storage type) and I’ll outline a tailored measurement plan.