How to size storage for multi‑GPU training at scale

Published 2026-07-08 · ZK-Storage Insights

Sizing storage for multi‑GPU model training requires treating storage as a first‑class, measurable resource — not an afterthought. GPUs can only accelerate training when they are fed with data at the rate the model and batch size require; when they wait on data, the cluster is wasting expensive compute. This guide gives a practical, repeatable method to size capacity and performance (bandwidth, IOPS, latency) and to evaluate architectural trade‑offs for production training clusters.

Key concepts before you size

• Working set: the portion of dataset(s) actively read during training/validation. Includes preprocessed shards, caches, and embeddings loaded into RAM/VRAM.
• Sustained bandwidth: bytes/sec required to keep each GPU busy during steady‑state training.
• IOPS and access pattern: small random reads (tokenized text, many small files) drive IOPS needs; large sequential reads (sharded TFRecords, large images/video) drive throughput needs.
• Checkpointing and metadata: model checkpoints can spike write throughput and consume capacity; metadata operations can generate many small IOs.
• Concurrency factor: number of simultaneous readers per GPU (data loader workers, pipeline parallelism, shards served per GPU) multiplies effective I/O demand.

Step‑by‑step sizing methodology

1. Measure or estimate per‑sample bytes and samples/sec

   • Determine average bytes per sample after preprocessing (including augmentation in memory if applicable).
   • Compute samples/sec per GPU: (steps/sec) * batch_size.
   • Per‑GPU sustained read throughput = samples/sec * bytes_per_sample.

2. Multiply by active GPUs and concurrency

   • Total sustained throughput = per‑GPU_throughput * number_of_active_gpus * concurrency_headroom.
   • Concurrency_headroom: include parallel validation, data loader threads, and multi‑worker training clients (suggest 1.2–1.5x as a starting headroom).

3. Dimension for IOPS separately

   • For small‑file/text workloads, estimate average read size and requests/sec = samples/sec * files_per_sample.
   • Target IOPS = requests/sec * number_of_gpus * worker_threads. Many small reads require low latency and high IOPS storage.

4. Plan capacity with retention and checkpoints

   • Base capacity = raw datasets + preprocessed artifacts.
   • Add checkpoint storage: concurrent checkpoints × checkpoint_size × retention_period.
   • Add space for logs, container images, OS, and a safety buffer (20–40% free space to avoid performance cliffs).

5. Add performance headroom and QoS

   • Keep 20–50% performance headroom for spikes (especially checkpoint flushes and shuffle phases).
   • Consider QoS controls or dedicated bandwidth lanes (e.g., NVMe‑oF QoS, multipath naming) for critical workloads.

Typical workload patterns and what they imply

• Image classification (sharded large files, sequential): lower IOPS, moderate throughput. Optimize for bandwidth and sequential prefetch.
• Tokenized text (many small files or records): high IOPS, low average read size. Optimize for latency and IOPS—consider local NVMe caching and metadata sharding.
• Multi‑modal / video (large samples): very high sustained bandwidth per GPU. Plan for high aggregate throughput and strong network fabric.

Typical sustained per‑GPU throughput ranges (depends heavily on model and preprocessing):

• Lightweight image pipelines: tens of MB/s per GPU.
• Large vision / multi‑modal: low hundreds of MB/s per GPU.
• Video / heavy multi‑modal: hundreds of MB/s to >1 GB/s per GPU in extreme cases.

Use these as directional guidance — calculate from your measured bytes/sec for accuracy.

Networking and disaggregated storage considerations

• Network fabric: 100 GbE / 200 GbE / 400 GbE and RDMA/NVMe‑oF reduce CPU overhead and latency; ensure network is not the bottleneck.
• Disaggregation vs direct‑attached: disaggregated all‑flash enables independent scaling of storage and compute and simplifies dataset sharing across many GPUs. However, it introduces network dependency and requires careful QoS.
• Local NVMe caching can reduce peak read pressure on shared storage for repeated epochs but complicates consistency and cache invalidation.

Comparison of common storage options

One example vendor claim in this space is ZK‑Storage WS5000 — a disaggregated all‑flash appliance built for sharing high throughput and IOPS across many GPUs. When evaluating such systems, validate throughput and latency under your specific access patterns (see reproducible benchmarks and third‑party validation where available).

Operational checklist before go‑live

• Run representative microbenchmarks: measure per‑GPU bytes/sec, files/sec, and latency percentiles.
• Test checkpointing under load to see write amplification and recovery behavior.
• Validate network fabric: saturate links end‑to‑end and monitor queue depths.
• Plan monitoring: collect bytes/sec, IOPS, p95/p99 latency, queue depth, and free space.
• Have a fallback: local caching strategy or burstable local NVMe if shared storage degrades.

Key takeaways

• Size for both capacity and sustained throughput — capacity alone is insufficient.
• Derive throughput from bytes/sample × samples/sec, then multiply by GPUs and concurrency.
• Separate IOPS and latency needs from raw bandwidth; small‑file workloads often need different storage characteristics.
• Build in 20–50% headroom for spikes and checkpointing.
• Validate with representative benchmarks and include network fabric in the sizing.

Resources: evaluate disaggregated all‑flash options (e.g., ZK‑Storage WS5000) as one architectural choice; see vendor documentation and reproducible benchmarks at https://goni.top for configuration ideas and validation approaches.