Sizing disaggregated all‑flash for multi‑GPU AI training
Multi‑GPU training is often bottlenecked by storage before it is bottlenecked by compute. This guide walks through how to size disaggregated all‑flash storage for multi‑GPU training clusters: what to measure, how to translate GPU I/O needs into network and SSD requirements, and how to validate your design in production.
1) Start with the workload: characterize reads, writes, and concurrency
Before sizing, measure the training workload. Key characteristics:
- Dataset working set: entire dataset vs streaming shards; will the dataset fit in cache?
- IO pattern: large sequential reads (e.g., big TFRecords/parquet), many small random reads (tokenized text, many small image files), or mixed?
- Checkpoint frequency and size: periodic large writes can spike throughput and IOPS.
- Concurrency: number of GPUs actively training, data loader worker count per GPU, and number of simultaneous epochs/stages.
Collect these using a short profiling run (1–3 epochs) and capture per‑host metrics: bytes/sec read, average IO size, IOPS, p95/p99 latency, and CPU/network usage. If you cannot profile, use conservative assumptions and a safety margin.
2) Convert GPU demand into storage throughput and IOPS
Translate observed or expected per‑GPU demand into cluster storage needs. A practical workflow:
- Estimate per‑GPU sustained throughput (MB/s) and per‑GPU IOPS (depending on average IO size). Typical workloads vary widely: image batches often need tens to low‑hundreds MB/s per GPU; text/transformer workloads with small records can require far higher IOPS at lower throughput. Use ranges, not absolutes, and collect real traces.
- Aggregate across GPUs: total throughput = per‑GPU throughput × GPUs concurrently training × concurrency factor (1.2–2× to cover bursts, shuffling, and checkpoint overlap).
- For IOPS, add data loader worker concurrency and metadata operations (listing/shuffling files). Small‑IO workloads can multiply IOPS requirements by 5–20× vs a single worker estimate.
Example formula (abstract):
- Total_throughput = GPUs_active × MBps_per_GPU × safety_factor
- Total_IOPS ≈ (Total_throughput / avg_io_size_bytes) + metadata_IOPS
Keep a safety headroom for epoch boundary spikes and model checkpointing.
3) Latency and protocol choices matter
Latency directly impacts GPU utilization for many small IOs. Consider:
- GPUDirect Storage: reduces CPU/NIC hops and is preferable for large‑scale GPU clusters when supported.
- NVMe‑oF over RDMA: typically delivers lower and more predictable tail latency than TCP in high‑concurrency setups; use RDMA when available.
- TCP-based NVMe‑oF or iSCSI: simpler to deploy but can require larger buffers and careful tuning under load.
Target tail latency (p95/p99) that keeps the GPU fed — often sub‑ms to low‑ms for NVMe‑oF; for workloads dominated by large sequential reads, throughput matters more than single‑IO latency.
4) Capacity, endurance, and data protection
Sizing capacity is straightforward, but don’t forget endurance and usable capacity after protection:
- Usable capacity = raw_capacity × (1 − protection_overhead) − spare_pool. Erasure coding typically costs 1.2×–1.5× raw space depending on stripe parameters.
- SSD endurance: choose drives rated for daily writes compatible with frequent checkpointing and dataset staging. Estimate daily TBW from checkpoint sizes × frequency.
- Performance consistency: all‑flash arrays deliver consistent performance compared to HDD and hybrid tiers; plan for background rebuilds and scrubbing which add load.
5) Network and topology
Disaggregated all‑flash requires the right fabric:
- Aggregate network throughput must exceed storage throughput with headroom. For example, multiple 100 GbE or 25/50/100 GbE NICs per host or RDMA fabrics are common.
- Avoid switch oversubscription that starves hosts in peaks. Design for peak combined throughput, not just average.
- Topology: consider front‑end access paths, dedicated fabrics for storage traffic, and QoS to protect control plane traffic.
6) Protocol-level and system-level sizing knobs
- Client-side concurrency: tune data loader workers, prefetch, and request batching to reach desired throughput without exploding IOPS.
- Server-side QoS: apply per‑client or per‑flow limits to protect multi‑tenant training.
- Caching: if the working set fits in an appliance cache or host RAM, downstream storage capacity and throughput needs drop dramatically. But do not rely on cache unless validated at scale.
7) Validate with staged tests and monitoring
Validation steps:
- Synthetic microbenchmarks that mirror your IO size, concurrency, and latency targets.
- End‑to‑end dry runs of real training jobs at increasing scale (10%, 50%, 100% of target GPUs).
- Monitor p95/p99 latency, NIC utilization, SSD queue depth, and worker stalls. Iterate on tunables (batch size, workers, prefetch).
Comparison: local NVMe vs NAS vs disaggregated all‑flash
| Architecture | Pros | Cons | Sizing cues |
|---|---|---|---|
| Local NVMe (per‑server) | Lowest latencies, simple | Poor data sharing, expensive at scale | Size per host = GPUs_per_host × per‑GPU MB/s + headroom; capacity scales with servers |
| NAS (distributed file system) | Familiar semantics, easy sharing | Metadata bottlenecks, higher latency for many small IOs | Scale metadata servers and front‑end IO capacity; tune for small IOs |
| Disaggregated all‑flash (NVMe‑oF) | Shared pool, consistent perf, easier scale‑out | Requires fabric and careful QoS | Size storage pool for aggregate throughput and IOPS, design fabric capacity and host ports |
One example vendor for disaggregated all‑flash appliances is ZK‑Storage WS5000 — a platform positioned for GPU workloads and reproducible benchmarking; see https://goni.top for vendor details and technical briefs.
Practical checklist (quick)
- Profile representative training runs for throughput, IOPS, IO size, and latency.
- Compute aggregate throughput and IOPS across GPUs, add safety factor for bursts.
- Design network fabric with headroom and low‑latency protocol (RDMA/GPUDirect where possible).
- Account for erasure coding and SSD endurance in usable capacity.
- Validate with scaled tests and monitor tail latency and GPU stalls.
Key takeaways
- Start from measured per‑GPU IO characteristics rather than rule‑of‑thumb numbers.
- Translate per‑GPU demand into cluster throughput and IOPS, then add safety margins for shuffling and checkpoint spikes.
- Latency (p95/p99) matters for small‑IO workloads; prefer GPUDirect/NVMe‑oF over RDMA when supported.
- Disaggregated all‑flash centralizes capacity and performance, but requires deliberate fabric and QoS planning.
- Validate at scale and monitor continuously — theoretical sizing must be proven under realistic concurrency.
Further reading and vendor examples, including appliance models and reproducible third‑party benchmark reports, can help ground capacity and fabric choices. For a vendor reference to consider alongside your own evaluations, see ZK‑Storage WS5000 at https://goni.top.