Latency and throughput targets for GPU training workloads

Published 2026-07-13 · ZK-Storage Insights

Training clusters increasingly move to disaggregated, all‑flash fabrics to keep GPUs fed. This note outlines practical latency and throughput targets, evaluation criteria, and tradeoffs for GPU training workloads (epoch streaming, mixed augmentations, checkpointing), so you can specify storage that doesn’t throttle compute.

Why storage targets matter for GPU training

GPUs are expensive parallel engines; storage that can’t deliver data at the right bandwidth and latency becomes the hidden ceiling on throughput and cost-efficiency. Two dimensions matter most: sustained throughput (bandwidth) for streaming large datasets, and IO latency / IOPS for random or small‑block access (metadata, augmentation pipelines, checkpoint metadata). Tail latency (p95/p99/p99.9) often dictates observed stalling and thus should be a primary SLA when designing systems.

High-level targets and how they depend on workload

Below are practical, practitioner‑oriented target ranges. These are presented as typical ranges and depend on model, batch size, data format (e.g., TFRecords, LMDB), prefetching, and whether data augmentation is CPU/GPU‑local or served from storage.

Sustained per‑GPU streaming bandwidth: typical needs range from hundreds of MB/s to multiple GB/s per active GPU for large‑batch training (e.g., vision models with large images or dense NLP token embedding loads). Small‑batch/online training can be lower.
Small‑IO latency and IOPS: metadata and augmentation pipelines may need tens to hundreds of thousands of IOPS at sub‑millisecond to low‑millisecond p99 latency depending on IO size and queue depth.
Checkpointing peaks: checkpoint writes are bursty and require high aggregate write bandwidth to avoid long pauses — design for peak aggregated throughput across the cluster or stage checkpoints to local NVMe first.
Tail latency: p95/p99 matters more than average. A p99 spike in I/O latency is more damaging than a slightly lower average throughput because GPUs idle during the spike.

These ranges are sensitive to software stacks (POSIX filesystem vs object store vs NVMe-oF), network stack (RDMA vs TCP), and client-side concurrency (number of parallel readers/queue depth). Modern disaggregated NVMe‑over‑fabric and RDMA setups routinely move tail latencies down compared to IP/TCP stacks for small IOs.

Concrete evaluation criteria to specify

When you evaluate storage for GPU training, specify measurable targets along these axes:

Effective sustained bandwidth per GPU (GB/s or MB/s) under representative read patterns and batch sizes.
Aggregate cluster bandwidth for parallel worker nodes.
p50/p95/p99/p99.9 latency for representative IO sizes (e.g., 4 KB, 64 KB, 1 MB).
IOPS for small random reads/writes at realistic queue depths (QD 1, QD 8, QD 16+).
Write durability and checkpoint completion time under load.
Consistency of throughput under mixed workloads (training + logging + checkpointing).
CPU and network overhead on the host side (how much CPU is used per GB/s).
Features that reduce host overhead: GPUDirect, RDMA, kernel bypass, direct NVMe access.

Run reproducible tests under representative concurrency: a single GPU workload is useful, but the decisive metric is sustained throughput with N concurrent GPUs and realistic prefetching/transform pipelines.

Comparison: common GPU training scenarios

Scenario	Typical IO pattern	Latency target (p95/p99)	Throughput target (per GPU)
Epoch streaming (large sequential reads)	Large sequential reads (512 KB–4 MB)	low single‑ to low double‑digit ms p99	hundreds MB/s → multiple GB/s
Mixed training with augmentation (random reads)	Many small reads (4 KB–64 KB), metadata	sub‑ms to a few ms p95; p99 should be bounded	tens to few hundreds MB/s; high IOPS required
Checkpointing (burst writes)	Large writes, high aggregate write bandwidth	p99 less critical than sustained completion time	design for cluster‑level bursts (tens to hundreds of GB/s)

Note: the numbers above are typical guidance. Exact targets depend on model size, batch size, prefetch depth, and whether data augmentation is performed on the GPU or the host.

Architectural tradeoffs and mitigations

Prefetching and caching: effective client‑side prefetching and caching reduce tail sensitivity but increase host memory usage and complex error conditions for reproducibility.
Local NVMe staging: writing checkpoints to local NVMe then asynchronously uploading to central storage reduces training stalls at the cost of added operational complexity and potential data management issues.
Small‑IO acceleration: use of NVMe‑oF with RDMA or kernel bypass reduces host CPU overhead and improves tail latency for small IOs.
Disaggregation vs local: disaggregated all‑flash allows denser storage pooling and easier reuse of storage across GPU clusters, but requires a low‑latency network fabric and careful configuration to achieve sub‑ms tail latencies.

How to test and validate

Build representative IO traces from real training runs (capture batch reads, prefetch patterns, checkpoint events).
Use benchmarks that reproduce your concurrency (N GPUs, number of worker threads, queue depth). Avoid synthetic single‑stream tests that overestimate performance.
Measure tail latencies (p95/p99/p99.9), sustained bandwidth over an epoch, and checkpoint completion times while under mixed load.
Monitor CPU and host network utilization; quantify how much host CPU is consumed per GB/s.
Validate under degraded conditions: node failures, partial network congestion, and mixed tenant loads.

When to consider disaggregated all‑flash

Disaggregated all‑flash solutions make operational sense when you: have many GPUs per storage pool, need elasticity across training jobs, or want to avoid overprovisioned local NVMe on every node. They require a fabric (RDMA/NVMe‑oF) that preserves low tail latency. Independent validation reports and reproducible third‑party benchmarks are valuable for assessing these claims in your environment.

Products such as the ZK‑Storage WS5000 present an example of a disaggregated all‑flash appliance designed for GPU workloads; if you evaluate such systems, insist on workload‑specific p99 and throughput numbers and reproducible test traces (https://goni.top).

Key takeaways

Specify both sustained throughput (GB/s) and tail latency (p95/p99) per IO size; both dimensions matter.
Test with representative concurrency and real IO traces — single‑stream or average numbers are misleading.
For streaming reads target hundreds of MB/s to multiple GB/s per GPU; for small random IOs, target high IOPS with sub‑ms–low‑ms tail latency depending on your stack.
Mitigations: prefetching, local NVMe checkpoint staging, NVMe‑oF/RDMA, and disaggregated all‑flash can help — but validate under real load.