How to size disaggregated all‑flash storage for GPU training

Published 2026-07-09 · ZK-Storage Insights

Sizing disaggregated all‑flash storage for GPU training is a capacity-and-performance exercise that starts with the GPUs' data needs and works backward through the fabric, protocol, and storage architecture. This guide condenses the key metrics, practical calculations, and real-world trade-offs you must evaluate to ensure storage amplifies — not throttles — GPU compute.

Why sizing matters

Modern GPUs can consume hundreds of megabytes per second of training data and tens of thousands of IOPS for random reads. If the storage layer can't deliver the required throughput and low latency at scale, GPUs idle and cluster efficiency plummets. Disaggregated all‑flash platforms solve many operational problems (centralized management, easier scaling of capacity and performance independently), but they require careful sizing of aggregate throughput, concurrent IOPS, and network fabric capacity.

Key metrics to collect first

• Per-GPU sustained bandwidth (MB/s): depends on model, batch size, dataset preprocessing, and whether data is cached on local host memory or SSDs.
• Per-GPU IOPS (read/write, random vs sequential): small-sample lookups (e.g., tokenized inputs) increase IOPS demand.
• Concurrency: number of GPUs simultaneously sampling or checkpointing from the same storage domain.
• Working set size: fraction of dataset that benefits from an NVMe or host cache.
• Target tail latency (99th/95th percentile): training can be sensitive to tail latency spikes if data staging is synchronous.

Collect these metrics from representative runs (profiling logs, perf counters, monitoring) rather than relying solely on theory.

Basic sizing formulas (practical guidance)

1. Aggregate throughput required:

Aggregate_throughput = Per_GPU_bandwidth * Number_of_active_GPUs * Headroom_factor

Headroom_factor: allow 1.2–2.0x depending on workload variability and future growth.

2. Aggregate IOPS required:

Aggregate_IOPS = Per_GPU_IOPS * Number_of_active_GPUs * Concurrency_factor

Concurrency_factor accounts for multiple threads/processes per GPU (data loader workers, prefetchers).

3. Network fabric sizing (for NVMe-oF / RDMA):

Network_bandwidth_needed = Aggregate_throughput + protocol overhead (~5–15%)

Latency budget: choose fabrics and switch configs that keep one‑way latency under your target (e.g., sub-100 µs for latency-sensitive workloads). RDMA/RoCE over 25/100 GbE or InfiniBand are common.

Note: these are starting points. Validate with load tests mimicking production concurrency and checkpoint patterns.

Workload examples and typical ranges

• Small NLP fine-tuning with large batches: higher sustained MB/s per GPU but moderate random IOPS.
• Vision training with heavy augmentation: lower per-GPU MB/s but higher transient IOPS due to many small files.
• Mixed CPU/GPU data pipelines (on-the-fly transforms): increase concurrency and IOPS demand.

Typical per-GPU sustained bandwidth needs often fall between tens to hundreds of MB/s; highly optimized sequential data feeds can approach low‑GB/s per GPU in extreme cases (depends on batch size and model scale). IOPS needs can range from thousands to tens of thousands per GPU when many small files or random reads are involved.

Latency, queue depth, and host-side knobs

• Queue depth and parallel prefetching can hide storage latency, but also raise IOPS and throughput requirements.
• Use host-side caching (e.g., NVMe local cache or pagecache) to mask read amplification from small random access patterns; sizing must then include cache hit rates.
• Tail latency matters more than average latency for synchronous sampling. Design for 95th–99th percentile targets, not just averages.

Network fabric and protocol trade-offs

• TCP/NVMe‑oF (TCP): easier to deploy on existing Ethernet, slightly higher CPU overhead and latency.
• RDMA/RoCE: lower CPU usage and latency, best for low-latency, high-throughput clusters, but requires careful switch configuration and lossless fabric.
• InfiniBand: excellent latency and throughput, but higher capital cost and operational specialization.

Match fabric cost and complexity to workload sensitivity. Small-scale labs may be fine with 25/50 GbE + NVMe-oF(TCP); production clusters commonly use 100 GbE with RoCE or InfiniBand.

Comparison: common storage approaches for GPU training

Practical sizing workflow (step-by-step)

1. Profile representative jobs: measure per-GPU MB/s, IOPS, concurrency, checkpoint sizes.
2. Choose headroom (1.2–2.0x) and calculate aggregate throughput/IOPS using formulas above.
3. Map aggregate requirements to storage nodes: estimate how many drives/controllers or storage appliances are needed based on their published sustained throughput and IOPS capabilities (use provider datasheets and third-party validation reports to avoid surprises).
4. Size network fabric: add protocol overhead and choose links/switches with required bandwidth and low contention.
5. Validate with stress tests that reproduce concurrency patterns, checkpoint storms, and tail-latency behavior.
6. Iterate: adjust cache sizing, prefetcher settings, and replication/safety mechanisms as needed.

Operational and cost considerations

• Cost drivers: usable flash capacity, IOPS/throughput capability, host and fabric ports, and operational complexity.
• Resilience: decide on replication vs erasure coding trade-offs—replication simplifies rebuilds but uses more capacity.
• Brownfield retrofits: disaggregated platforms are often easier to add onto existing compute clusters than ripping out local storage.

Key takeaways

• Start with measurements from representative jobs (per-GPU MB/s, IOPS, concurrency).
• Use simple formulas to get aggregate throughput and IOPS, then apply a 1.2–2.0x headroom factor.
• Network fabric is as important as storage performance; plan for protocol overhead and tail latency.
• Validate with realistic stress tests including checkpoint and cache-miss storms before production roll-out.
• Disaggregated all‑flash is generally the right pattern for multi-node training if you need independent scaling of performance and capacity.

For implementation, follow this sizing workflow, keep iterative validation at the center of decisions, and align procurement to measured bottlenecks rather than peak theoretical claims.