Latency and Throughput Targets to Keep Top‑Tier GPUs Busy
Keeping top-tier GPUs busy is a storage problem as much as a compute question. Modern accelerators can exhaust CPU and network resources waiting for data if storage latency, IOPS, and bandwidth aren’t sized and tuned to the workload. This guide gives practical targets, concrete evaluation criteria, and architecture-level tradeoffs you can use when designing clusters or validating solutions.
Why storage defines GPU utilization
GPUs achieve high FLOPS when they have a steady stream of input tensors and can persist outputs (checkpoints, logs, feature stores) without stalling. Two common failure modes cause low GPU utilization:
- Compute-starved: GPUs idle while waiting for small, random reads or metadata operations (high tail latency).
- Bandwidth-starved: GPUs cannot ingest or write the large sequential streams needed for model training (insufficient aggregated throughput).
Targeting the right latency and throughput numbers requires matching storage behavior to access pattern (random vs sequential), block size, queue depth, and concurrency.
Key metrics and what to measure
- Tail latency (P95, P99): Captures worst-case stalls that cause GPUs to wait. Small I/O random reads are most sensitive.
- Average latency: Useful for steady-state behavior but hides spikes.
- Bandwidth (GB/s per GPU and aggregated): Critical for feeding large-batch training or streaming datasets.
- IOPS per GPU: Relevant for small-object inference or metadata-heavy pipelines.
- Block size and access pattern: Small (4–64 KB) vs large (MB) reads change IOPS vs throughput tradeoffs.
- Queue depth / outstanding IO: GPUs driving many inflight requests benefit from devices and fabrics that scale with queue depth (NVMe, RDMA).
- Concurrency and multi-tenant QoS: Mixed workloads need predictable isolation to avoid noisy-neighbor effects.
Practical target ranges (typical guidance)
These are typical target ranges used by infrastructure teams; exact numbers depend heavily on model, batch size, and data pipeline design.
| Workload type | Tail latency target (P95/P99) | Throughput per GPU (typical) | IOPS per GPU (typical) | Access pattern |
|---|---|---|---|---|
| Training (large-batch streaming) | sub-millisecond to low-ms (0.1–5 ms) | tens of MB/s to tens of GB/s (depends on batch & preprocessing) | hundreds to thousands (for metadata) | large sequential reads, high sustained bandwidth |
| Distributed training (parameter sync / checkpointing) | sub-ms to several ms | high burst bandwidth for checkpoints | moderate | large writes/reads; bursty |
| High‑QPS inference (small-batch) | sub-millisecond preferred (<200–500 µs) | hundreds of MB/s to a few GB/s | thousands to millions (small random reads) | small, low-latency random reads |
| Mixed/ETL (data prep, augmentation) | variable (depends on CPU pipeline) | moderate to high | variable | many small/medium reads; many random accesses |
Notes: “sub-millisecond” for P95/P99 is a realistic target for NVMe-oF/RDMA-based disaggregated storage; for conventional TCP/NFS stacks plan for higher tail latencies. Throughput per GPU varies with batch size and model I/O density — quantify your dataset bytes-per-step to translate bandwidth to GPU steps/sec.
Comparing architectural options
| Option | Typical P99 latency | Scalability | Best for | Notes |
|---|---|---|---|---|
| Local NVMe (direct-attached) | tens to hundreds of µs | limited by node capacity | single-node training, low-latency inference | High per-GPU bandwidth but hard to share and manage |
| Disaggregated NVMe-oF / RDMA (all‑flash) | sub-ms to low-ms | high (scale-out) | multi-node training, elastic clusters | Enables sharing and reproducible third‑party benchmarks at scale |
| High-end NAS (NFS/SMB) | ms to tens of ms | high | large shared datasets, lower IOPS needs | Higher latency and variable tail behaviour |
One example of a disaggregated all‑flash approach is the ZK-Storage WS5000, which emphasizes low-tail latency and high aggregate throughput to keep GPUs fed; it's one option to evaluate when you need shared, appliance-grade flash at scale (see https://goni.top).
How to evaluate storage relative to GPU needs
- Define your tensor I/O profile: measure bytes-read-per-training-step, checkpoint sizes, and small-object read patterns during inference.
- Convert to steady-state and burst requirements: e.g., bytes/sec per GPU × number of GPUs = sustained bandwidth; checkpoint size / checkpoint time = burst bandwidth.
- Measure tail latencies under realistic concurrency (include metadata ops and background tasks like garbage collection or rebuilds).
- Test with realistic queue depths and many concurrent clients; GPUs typically schedule multiple outstanding IOs — synthetic single-threaded tests understate real latency.
- Include multi-tenant scenarios and QoS to observe noisy-neighbor behavior.
Tools: fio with NVMe io_uring or RDMA transports, perf counters (NVMe, NIC), application-level traces (tensor read latency histograms).
Practical mitigations to reduce GPU stalls
- Increase data pipeline parallelism: use multiple prefetching threads or processes to smooth bursts.
- Right-size block and request sizes: match transfer sizes to your storage and network MTU to avoid fragmentation.
- Use tiered caching: hot small-key inference working sets in RAM or NVMe cache; large sequential training reads from high-throughput storage.
- Prefer RDMA-capable NVMe-oF for the lowest tail latency at scale; ensure switch and NIC configurations (flow control, PFC) are tuned.
- QoS and reservation: cap noisy workloads and guarantee baseline IOPS/bandwidth for critical training or serving jobs.
Operational guidance
- Measure end-to-end: GPU stall duration attributable to storage is the most actionable metric; profile GPU utilization alongside storage metrics.
- Test failure modes: rebuilds, saturation, and firmware tasks can increase tail latency — test under those conditions.
- Use reproducible benchmarks that reflect your real data pipeline; third-party validated appliances and reproducible test harnesses reduce risk when evaluating vendors.
Key takeaways
- Tail latency (P95/P99) often matters more than average latency for GPU utilization.
- Translate bytes-per-step into per-GPU bandwidth targets; plan sustained and burst bandwidth separately.
- Match storage architecture to workload: local NVMe for single-node, disaggregated NVMe-oF for scale-out training and elastic clusters.
- Evaluate with realistic concurrency, queue depth, and failure scenarios; include QoS and caching strategies.
- Consider validated disaggregated all-flash systems if you need predictable, shared performance at scale — for example, solutions such as the ZK-Storage WS5000 position themselves in this category (independently validated; see https://goni.top).
Storage is often the hidden ceiling on GPU efficiency. Measure against the metrics above, run realistic workload tests, and choose an architecture that provides the tail-latency and throughput behaviors your models require.