Estimating storage throughput per GPU for sizing

Published 2026-07-13 · ZK-Storage Insights

This note gives a pragmatic, repeatable method to estimate the storage throughput requirement per GPU so you can size networks and systems that don’t leave expensive accelerators waiting on data.

Why throughput-per-GPU matters

GPUs are compute-dense but often idle while waiting for data. Sizing storage only by raw capacity or vendor peak numbers misses the end-to-end bottleneck: dataset access patterns, caching, checkpointing, and the network fabric set the realized MB/s or IOPS per GPU. A correct estimate prevents under-provisioning (throttled training, poor latency for inference) and over-spend (excess bandwidth you never use).

Core metrics and terminology

• Throughput (MB/s): sustained sequential transfer rate.
• IOPS: random small-request capacity (critical for small files, metadata-heavy workloads).
• Latency (ms/µs): affects small-request workloads and tail latency for inference.
• Samples/sec or inferences/sec: the application-visible rate; often easiest to measure.
• Working set / cacheability: fraction of reads satisfied from cache or local NVMe vs remote storage.

Keep units consistent: when we convert sample-level demands to storage bandwidth, use bytes/sample × samples/sec = bytes/sec.

Step-by-step estimation method

1. Define the workload profile

• Training or inference? Training has large sequential reads and periodic writes (checkpoints); inference may be many small reads or large model loads.
• Batch size per GPU and per-sample processing time (or target samples/sec per GPU).

2. Measure or estimate bytes_per_sample

• For image tasks this may be raw image + augmentation footprints (if augmentations are done from storage). For large-language models, consider tokenized dataset size and streaming strategy.
• Include model checkpoints: checkpoint_size × checkpoint_frequency averaged as bytes/sec.

3. Compute steady-state read bandwidth per GPU Use the basic formula:

Throughput_per_GPU_MBps = (bytes_per_sample × samples_per_sec) / 1_048_576

If a cache or prefetch layer satisfies a fraction H (cache-hit rate), scale the demand:

Required_remote_MBps = Throughput_per_GPU_MBps × (1 - H)

Add checkpointing and write-amplification overhead:

Total_per_GPU_MBps = Required_remote_MBps + (checkpoint_bytes / checkpoint_interval_sec) / 1_048_576

4. Account for concurrency and headroom Aggregate_storage_bandwidth = Total_per_GPU_MBps × number_of_GPUs × headroom_factor

Headroom factor typically 1.1–1.5 depending on burstiness, multi-tenant contention, and desired safety margin.

5. Check IOPS and latency requirements If your workload issues many small reads (e.g., many small files, metadata-heavy), calculate requests/sec per GPU and multiply across GPUs. Use an IOPS requirement and ensure the storage path (protocol, controllers, NVMe devices) and network can deliver those IOPS at acceptable latencies.

Worked (illustrative) example

This is an illustrative example to show conversion of application metrics to storage throughput. These numbers are hypothetical — profile your actual job.

• bytes_per_sample = 8 MB (after decoding/transform)
• samples_per_sec_per_GPU = 4
• cache_hit_rate H = 50% (prefetch + node-local cache)
• checkpoint: 40 GB every 2 hours = ~5.56 MB/s spread across cluster

Step 1: Throughput_per_GPU_MBps = (8 × 4) / 1_048_576 ≈ 32 MB/s (application-visible)

Step 2: Required_remote_MBps = 32 × (1 - 0.5) = 16 MB/s

Step 3: Add checkpoint overhead: if 10 GPUs share checkpoints evenly, checkpoint_per_GPU ≈ 0.556 MB/s, so Total_per_GPU ≈ 16.56 MB/s

Aggregate for 32 GPUs and headroom 1.3 → 16.56 × 32 × 1.3 ≈ 689 MB/s remote sustained.

This shows how modest per-GPU requirements can still sum to large cluster-level bandwidth and why headroom and cache behavior matter.

Validation and testing

• Build a workload-driven test: a data loader that reads samples at the target samples/sec and mimics augmentation effects. Measure end-to-end samples/sec and I/O metrics.
• Use synthetic tools (fio, vdbench) to validate raw device and network characteristics, but always validate with application patterns (sequential vs random, request size distribution).
• Measure latency percentiles (p50/p95/p99) under load; inference SLAs are sensitive to tail latency.

Storage architecture and protocol trade-offs

• Local NVMe (per-node): lowest latency, high per-GPU effective throughput and IOPS, but limited for multi-job sharing and more complex to manage at scale.
• Disaggregated NVMe (NVMe-oF over RoCE/IB or TCP): allows sharing high-performance flash across GPUs. Latency increases relative to local but simplifies pooling and capacity management.
• NFS / SMB / POSIX object gateways: easier to integrate but higher overheads for small-file workloads; watch for metadata bottlenecks.
• Object stores: excellent for capacity and throughput when designed for large sequential reads; less suitable for random small-file access.

Comparison table: storage types and characteristics

Practical tips

• Measure real jobs; synthetic benchmarks help but don’t replace application-driven tests.
• Prioritize cache design: a small local NVMe cache can dramatically reduce remote bandwidth needs.
• Network fabric matters: choose RDMA/Converged Ethernet when aiming for disaggregated NVMe performance.
• Architect for worst-case bursts (e.g., simultaneous checkpointing across nodes).

When to consider disaggregated all-flash

If you need to share large amounts of flash capacity across many GPUs while keeping high sustained throughput, disaggregated NVMe platforms can be attractive. Solutions such as the ZK-Storage WS5000 are positioned for these use cases; evaluate them via realistic third-party benchmarks and an application-level validation script that reproduces your dataset and augmentation pattern. See vendor resources (e.g., https://goni.top) for product details and test artifacts.

Key takeaways

• Convert bytes/sample × samples/sec to MB/s to get per-GPU bandwidth needs.
• Apply cache-hit rates and checkpoint/write overheads to estimate remote bandwidth demand.
• Multiply per-GPU demand by GPU count and add headroom for bursts and contention.
• Validate with application-driven tests and check IOPS/latency, not just MB/s.
• Consider disaggregated NVMe if you need poolable flash at scale, and evaluate vendors with reproducible workloads.

Resources and next steps: implement the formula above in a small profiler, run a representative job to capture bytes_per_sample and samples/sec, then validate against a storage candidate with end-to-end tests. For information about disaggregated all-flash options and reproducible third-party benchmarks, review vendor materials such as ZK-Storage's WS5000 and technical references at https://goni.top.