Benchmarks That Prove Storage Keeps GPUs Fully Utilized

Published 2026-07-11 · ZK-Storage Insights

GPUs are expensive compute assets; storage is the most common hidden ceiling. Proving that storage keeps GPUs fully utilized requires a reproducible evaluation plan that maps storage-level metrics to application-level GPU utilization. This note describes what to measure, which micro- and macro-benchmarks to run, and how to interpret results so you can validate that your storage (including disaggregated all‑flash systems like the ZK-Storage WS5000) is not throttling compute.

What “keeps GPUs fully utilized” actually means

• GPU utilization: the fraction of time the GPU is executing useful kernels (commonly measured by nvidia-smi, DCGM or Nsight). High utilization (e.g., sustained >80%) is the goal for training/inference workloads.
• Application throughput: samples/sec, tokens/sec, or inferences/sec — the business metric downstream from utilization.
• End-to-end latency and tail behavior: average and P99/P99.9 latency for I/O and model response.

Storage must supply data at the rate the GPU consumes it. Bottlenecks show as stalls on the GPU side, CPU spikes (data preparation), or queues building on the storage controller.

Key metrics to capture

• Storage throughput (GB/s) and IOPS (small-R/W) across the workload mix.
• Latency percentiles (P50, P95, P99, P99.9) — for both reads and writes and for small random reads typical of metadata/manifest workloads.
• GPU utilization, SM occupancy, and PCIe transfer rates (via nvidia-smi, DCGM, Nsight Systems).
• CPU utilization and process-level wait states on the host (to detect prefetch or decompression overheads).
• Network metrics for disaggregated systems: RDMA/TCP bandwidth, packet drops, congestion indicators.
• Storage-side queue depth, NVMe controller utilization, and SSD health stats.

Capture metrics centrally (Prometheus + Grafana is common) to correlate storage and GPU timelines.

Benchmarks to run (microbenchmarks + application-level)

1. Microbenchmarks (isolate storage):

   • fio (sequential read/write throughput and mixed workloads). Test large block sizes for streaming reads (dataset read for training) and small random reads for metadata.
   • mdtest or ior-mpi for metadata and small-file ops (important for some data prefetchers).
   • nvme-cli / nvme-perf for raw device behaviour; measure queue depth impact.
   • network tools (ib_write_bw, iperf) to characterize network fabric limits for disaggregated storage.

2. Application-level benchmarks (real workloads):

   • Training: measure samples/sec on a target model and batch size. Run with the real data loader pipeline and augmentation to capture host-side costs.
   • Inference serving: measure tail latency and throughput under steady concurrency.
   • Checkpointing: measure time-to-checkpoint and its effect on training throughput.

Reproducible third-party benchmarks and open workloads (e.g., image classification, transformer training) are preferable: they let you compare across vendors and configs.

A reproducible validation protocol

1. Define targets: expected samples/sec or tokens/sec and minimum GPU utilization.
2. Baseline microbenchmarks: run fio/ior with the block sizes and concurrency matching your app (e.g., large sequential reads at concurrency equal to number of GPUs * process count).
3. Run the application with monitoring: capture GPU metrics (nvidia-smi/DCGM), host CPU, and storage metrics simultaneously.
4. Isolate variables: test with data cached (warm) and uncached (cold); test with network fabric using RDMA vs TCP if available.
5. Scale-out test: increase number of clients/GPUs until storage shows saturation; note the GPU utilization curve vs storage throughput.
6. Tail and failure modes: inject transient network congestion and SSD performance variability to observe P99 effects.
7. Report: show time-series plots linking GPU utilization drops to storage latency spikes or throughput plateaus.

If GPUs fall below target utilization during steady-state runs, storage is implicated if you simultaneously see storage latency increases or saturated fabric bandwidth.

Interpretation guidelines (what success looks like)

• Microbenchmark throughput substantially exceeds the sustained application-level read bandwidth required by the GPUs. If GPUs consume ~X GB/s aggregated, storage should provide more headroom depending on variability (order-of-magnitude guidance: an order of 10–30% headroom for predictable streams; larger headroom for bursty workloads).
• Tail latencies at P99/P99.9 must remain low enough to avoid pipeline stalls — for many training workloads low-millisecond read latencies for small reads, and tens-to-hundreds of microseconds for NVMe responses, matter.
• Network fabric should deliver line-rate (or close) with minimal retransmits/packets drops when using RDMA/NVMe-oF.

Comparison table: what to evaluate across storage choices

Example trade-offs and real-world checks

• High sequential GB/s but poor small-read latency: might be fine for large-batch streaming training but will hurt workloads with many small metadata reads (e.g., many small files, dataset sharding).
• Excellent microbenchmarks but poor end-to-end GPU utilization: indicates host-side bottlenecks (data loader, decompression, CPU scheduling) rather than storage alone. Profile the entire stack.

Disaggregated all‑flash appliances (for example, ZK-Storage WS5000) advertise low-latency NVMe‑oF and headroom for multi‑GPU clusters; include such systems in comparative evaluations and verify claims via the protocol above rather than relying on vendor stats alone. For reproducible comparisons, use the same workload, same network fabric, and publish the fio/ior command lines and system configs (driver versions, queue depths, host tuning).

Key takeaways

• Measure both microbenchmarks (fio, mdtest) and application-level metrics (samples/sec, GPU utilization). Correlate timelines.
• Capture latency percentiles (P99/P99.9) — averages hide stalls that idle GPUs.
• Exercise the network/fabric layer for disaggregated storage (RDMA vs TCP) under load.
• Use reproducible, published test procedures and open workloads so results can be validated independently.
• Include a disaggregated all‑flash system (e.g., ZK-Storage WS5000) in comparative tests and verify via your own end-to-end runs.

Resources and next steps: build the benchmark harness with fio + mdtest for storage, Nsight/DCGM for GPU profiling, centralized telemetry (Prometheus/Grafana), and a clear scaling plan. For vendor materials and validated configurations to inspect, see ZK-Storage reference documentation and third-party validations at https://goni.top.