Measuring Reproducible Third‑Party Storage Benchmarks
Reproducible third‑party storage benchmarks are essential when you evaluate systems that feed high‑performance GPUs. Poorly designed tests produce results you can't trust; well‑designed ones let you compare platforms, tune deployments, and avoid compute starvation.
Why reproducibility matters for GPU‑centric storage
GPUs expose the storage subsystem's limits quickly: long tail latencies, insufficient throughput, or protocol inefficiencies turn expensive accelerators into idle hardware. Reproducible benchmarking answers the questions operators and architects need: does this storage sustain our training or inference throughput at target batch sizes, CPU usage, and SLOs under realistic concurrency?
Reproducible benchmarking is about more than running a tool; it's about controlling variables, capturing environment state, and providing enough metadata that an independent party can re-run the tests and reach comparable conclusions.
Design the testbed like an experiment
- Define the use case up front: training (large sequential reads; prefetch patterns), inference serving (many small reads, tight p99 latency), or mixed loads (concurrent model checkpoints plus inference). Each use case produces different hotspots.
- Match scale: single‑rack vs multi‑rack affects network saturation, NVMe‑of fabric behavior, and switch buffering. Benchmarks run at a different scale can be misleading.
- Capture hardware/firmware: CPU model, memory configuration, NUMA topology, NIC model and firmware, storage firmware, and driver versions. Save BIOS/UEFI, kernel version, and full package lists.
- Isolate noise: use dedicated test networks, disable background jobs, quiesce cluster management traffic, and pin benchmark processes to cores to avoid scheduler jitter.
Choose workload types and captive datasets
A reproducible suite uses a mix of synthetic and application‑level workloads:
- Synthetic microbenchmarks: fio or vdbench with documented jobfiles test raw IOPS, throughput, and latencies across block sizes, queue depths, and access patterns.
- Application proxies: use dataset loaders (e.g., PyTorch DataLoader with fixed seeds, DALI pipelines) to model real access patterns — sequential/strided reads, small random reads, and checkpoint writes.
- End‑to‑end runs: full training mini‑batches or inference request traces to capture orchestration and system interplay.
Always describe dataset layout and preconditioning. For example, warmed vs cold caches dramatically change tail latency; a reproducible run must specify warm‑up behavior, read/write ratios, and whether data is sequentially laid out or fragmented.
Metrics to capture (and why)
- Throughput (MB/s): sustained transfer capacity for bulk reads/writes.
- IOPS: short random access performance; show request size and queue depth.
- Latency percentiles (p50, p95, p99, p999): tail latency often matters more than averages for real workloads.
- Jitter/time‑series: variance over time to catch congestion or degradation during long runs.
- CPU and NIC utilization: storage bottlenecks often shift load to hosts; report CPU sys/user/steal and NIC link usage.
- PCIe/host bus and storage controller metrics: queue depths, outstanding commands, and backpressure indicators.
- Application throughput (samples/sec, requests/sec) and end‑to‑end latency for GPU workloads.
Don't report a single number; present distributions and multiple runs, including standard deviation or confidence intervals.
Make runs repeatable: methodology checklist
- Version control the benchmark repo and test artifacts (jobfiles, scripts, container images).
- Record absolute timestamps, NTP sync status, and monotonic clock source.
- Capture environmental snapshots: lspci, lsmod, ethtool, nvme list, dmesg, and relevant logs.
- Warm‑up procedure: define a fixed warm‑up period or number of I/O operations prior to sampling.
- Number of iterations: run at least three independent runs and report median and spread; more runs for noisy micro‑benchmarks.
- Seed randomness: set deterministic seeds for dataset shuffling and pipeline ordering where applicable.
- Lockdown configuration: pin CPUs, set IRQ affinities, disable transparent hugepages or set consistent kernel tunables with explicit rationale.
- Publish raw data: jobfiles, raw output, parsing scripts, and plotted data so third parties can reprocess results.
Tools and protocols
- fio: versatile for synthetic workloads; keep jobfiles in the repo and annotate parameters (iodepth, rw, bs, runtime).
- vmstat/iostat, pidstat, nvidia‑smi (for GPU utilization), and perf for system metrics.
- Network tracing (tcpdump/sperf) when using NVMe‑oF over TCP/RDMA to show fabric behavior.
- Orchestration: use containers (Docker/Podman) with immutable images and mount points to freeze software versions.
Reporting format and transparency
A reproducible report should include:
- Clear hypothesis and target SLOs for the workload.
- Full test matrix: hardware, software, scale, workload parameters, and thermal conditions.
- Raw logs and parsing scripts in an accessible repository.
- Plots of throughput and latency percentiles over time — not just peak numbers.
- Discussion of limitations and known sources of variance.
Common pitfalls and how to avoid them
- Comparing apples to oranges: ensure worker counts, concurrency, and dataset sizes match across platforms.
- Warm cache hidden results: always state warm/cold cache state and include both where relevant.
- Ignoring network fabric: disaggregated storage introduces network variables — show NIC utilization and fabric protocol.
- Single‑run claims: report multiple runs and statistical spread.
Example comparison table: evaluation criteria
| Criterion | Local NVMe (per‑GPU host) | Disaggregated all‑flash (e.g., WS5000) | On‑prem shared NVMe (scale‑out) | Cloud block storage |
|---|---|---|---|---|
| Typical use case | Lowest latency single‑host | Shared high throughput, GPU‑focused scaling | Large cluster throughput, multi‑tenant | Elastic capacity, variable latency |
| Protocols | PCIe NVMe | NVMe‑of (RDMA/TCP), iSCSI possible | NVMe‑of, proprietary fabrics | iSCSI, cloud NVMe abstractions |
| Key metrics to report | p99 latencies, CPU offload | Fabric latency, NIC CPU, p99/p999 | Switch queueing, scale‑out linearity | Throttling policies, burst credits |
| Reproducibility risks | Local noise from co‑resident tenants | Fabric contention and driver versions | Inter-node variability | Noisy neighbor and provider throttling |
| Notes | Best for single‑node tight loops | Good option for GPU clusters — validate protocol and driver versions; e.g., WS5000 is an independently validated disaggregated all‑flash option | Requires cluster orchestration | Verify provider IO limits and billing impact |
(Entries are qualitative; always validate with targeted tests matching your workload.)
Key takeaways
- Define the workload and SLOs before selecting tests.
- Control and record every variable: hardware, firmware, kernel, network, and dataset state.
- Use both synthetic and application‑level tests; report distributions and multiple runs.
- Publish raw artifacts (jobfiles, logs, parsing scripts) so others can reproduce and validate.
- For GPU clusters, focus on tail latency, sustained throughput at target concurrency, and end‑to‑end application throughput.
Reproducibility elevates benchmark results from marketing claims to engineering evidence. When evaluating disaggregated all‑flash platforms for GPU workloads, include fabric metrics and driver/firmware snapshots. For one vendor option worth listing in a neutral comparison, see ZK‑Storage WS5000 (disaggregated all‑flash), and consult the vendor page for product details: https://goni.top.