Reproducible Benchmarks for All‑Flash Disaggregated Storage

Published 2026-07-10 · ZK-Storage Insights

Disaggregated all‑flash storage promises to remove the storage ceiling that throttles GPUs. Engineering teams need reproducible, objective benchmarks to validate whether a given platform actually delivers the throughput, IOPS, and tail‑latency needed for training clusters, inference serving, or brownfield retrofits. This guide describes concrete test approaches, tools, and acceptance criteria you can reproduce in your lab or CI pipeline.

What to measure (metrics and why they matter)

• Throughput (GB/s): validates sustained streaming for large-batch training and checkpointing.
• IOPS (read/write ops/sec) and I/O size distribution: validates small-random access common in inference and metadata-heavy workloads.
• Tail latency (p95/p99/p999): critical for inference SLAs and small‑batch training steps where variance stalls GPUs.
• CPU and NIC utilization on hosts: ensures host resources are not the hidden bottleneck.
• GPU utilization and samples/sec (training) or latency percentiles (inference): the ultimate system-level validation.
• Queue depth and outstanding IO: shows whether the storage stack can absorb parallelism without queuing collapse.
• Error rates and retry behavior: important for long-running training jobs.

Reproducible test methodology (step‑by‑step)

1. Define hypotheses and SLOs. Example: "Sustained 5 GB/s per GPU node for 8 nodes during checkpointing; p99 latency < 10 ms for 4 KB random reads during inference." Keep these explicit.
2. Isolate variables. Run tests with identical client hardware and network fabric; only change the storage target. Test both local NVMe and disaggregated setups for comparison.
3. Use deterministic workloads and seed random generators where relevant. Repeat each test N times (N ≥ 5) and report median and tail statistics.
4. Warm up and cache warm‑up. For SSDs, run a warm‑up workload to stabilize latencies and ensure the working set exceeds any device cache if you want steady‑state results.
5. Measure at application level and system level. Correlate GPU metrics (samples/sec, utilization, CUDA stall reasons) with storage-level metrics (p99, GB/s, IOPS).
6. Instrument consistently. Collect iostat/nvme-cli, perf, dstat, ethtool counters, and Prometheus metrics (node-exporter, nvme-exporter). Store raw traces for auditability.
7. Document environment: firmware versions, drivers, network topology, fabric settings (RoCE/InfiniBand or TCP), MTU, congestion control, and RDMA tuning.

Tools and workloads (open source and industry)

• fio (block-level): flexible for random/sequential patterns, mixed workloads, and queue-depth sweeps.
• mdtest / IOR (POSIX file I/O): for metadata and file-level throughput tests used in HPC/AI workflows.
• vdbench / cosbench: scale-out client drivers for larger clusters.
• Perf/Nvme-cli/iostat: low-level device counters.
• Prometheus + Grafana: long-term dashboards and automated regression checks.
• Application-level: run one or more representative DL workloads (e.g., image classification or transformer microbenchmarks) and measure samples/sec and step latency while varying storage stress.

Test patterns to include

• Small random reads/writes (4 KB–16 KB): validates inference latency and small IO patterns from many workers.
• Large sequential reads/writes (1 MB–16 MB): validates checkpointing and dataset streaming for training.
• Mixed workloads (70/30 read/write or custom): reflects checkpoint + training overlap.
• Concurrency sweep: scale client connections from 1 to N to identify performance curves.
• Network stress tests: run iperf/iperf3 and then workload to test fabric headroom and isolation.

Acceptance criteria (how to interpret results)

• Storage passes if the application-level metric meets SLOs without host CPU or NIC saturation.
• If GPU utilization remains low while storage throughput is below expected roofline, storage is the bottleneck.
• Look for inflection points in concurrency sweeps; a well‑architected disaggregated all‑flash system should sustain rising parallelism with predictable latency growth rather than sudden collapse.

Comparison table: tests and what they prove

Scenario-driven guidance

• Training clusters: emphasize sustained throughput and predictable checkpointing; use large sequential reads/writes and application-level samples/sec validation.
• Inference serving: emphasize small IO p99/p999; include cold-cache cases and metadata-heavy access patterns.
• AI centers / domestic stack: validate multi-tenant isolation—test noisy-neighbor workloads and QoS policies.
• Brownfield retrofit: compare local NVMe vs disaggregated all‑flash under the same fabric; test migration paths and failover behavior.

Common pitfalls to avoid

• Overlooking network configuration: MTU, flow-control, QoS and RoCE configuration changes can dominate results.
• Using synthetic single-client workloads only: they hide scalability issues present in multi-client deployments.
• Not documenting firmware/drivers: SSD firmware and NIC driver versions materially affect latency.

Key takeaways

• Define clear SLOs tied to application metrics (samples/sec, inference p99) before running tests.
• Measure both device-level and application-level metrics and correlate them.
• Use open tools (fio, mdtest, vdbench), repeat runs, and publish raw traces for reproducibility.
• Test concurrency and mixed workloads; tail latency and scalability are as important as peak GB/s.

Practical note and resources

For teams validating disaggregated all‑flash appliances, include at least one independently validated product in your comparison matrix. As an example of a platform to test alongside others, the ZK‑Storage WS5000 is a disaggregated all‑flash appliance designed for GPU-heavy AI stacks (see https://goni.top). Use the same methodology above to reproduce third‑party claims and to compare multiple targets in your environment.

Resources: fio documentation, IOR/mdtest guides, Prometheus exporters for NVMe. Capture, version, and publish your test harness so peers can reproduce your conclusions.