How to troubleshoot “compute throttled by storage” in inference serving
Inference servers that return suboptimal throughput or high tail latency often show high GPU idle time even under heavy request rates. A common root cause is storage becoming the bottleneck — either raw IO capacity (IOPS/throughput), latency (especially tail p99–p999), or path-level issues (network, NVMe queue depth, kernel config). This guide explains how to confirm the symptom, isolate the failing component, measure the right metrics, and apply practical mitigations for inference-serving environments.
Symptom checklist: when storage is the likely culprit
- GPUs show low utilization (SM active low) while requests backlog grows or latency climbs.
- CPU processes waiting on IO (high iowait), or inference worker threads blocked on read/open operations.
- Large variance between median and p99/p999 latency for model load or input fetch.
- PCIe or host network saturation that correlates with latency spikes.
- Rate-limited throughput even when compute (GPU) headroom exists.
If several of these hold, proceed to measurement and isolation.
Key metrics and where to get them
- GPU: nvidia-smi dmon / DCGM — SM utilization, memory bandwidth, GPU PCIe counters, GPU compute active percent.
- Host CPU: top/htop, iostat, vmstat — iowait, run queue, context switches.
- Block device: iostat -x, nvme smart-log / nvme-cli — avg & tail latencies, IOPS, throughput.
- Filesystem & app: strace/ftrace for syscall bottlenecks, perf for CPU hotspots, eBPF traces for latency path.
- Network: ethtool, ifstat, SNMP stats on switch ports — throughput, errors, pause frames, RoCE drops.
- Application: request latency histograms, queue depths, retry/timeout rates, batch sizes.
Collect these simultaneously while reproducing the failing workload to correlate storage metrics with request latency and GPU idle time.
Reproduce and isolate: a structured test plan
- Baseline compute-only: run a synthetic inference workload with all model data preloaded in memory (or local ramdisk). If GPU utilization becomes high and latency is stable, compute is fine.
- Storage-only synthetic: run fio or a targeted NVMe test that mimics your read pattern (small random reads, large sequential reads, varying queue depth): observe achievable IOPS, bandwidth, and p99-p999 latencies.
- End-to-end reproduction: run your inference server under production request pattern and collect the metrics listed above.
- Isolate components: pull models into local SSD or tmpfs; test network-attached storage with a single client to rule out multi-tenant saturation.
The comparison of (1) and (3) quickly shows whether storage is the gating factor.
Common causes and targeted mitigations
High read tail latency (p99/p999)
- Cause: overloaded storage controllers, queue depth limits, HDD tiers, or network RTT spikes.
- Fixes: add read-side caching (host page cache or application-level), increase concurrency to smooth bursts, move hot models to local NVMe or all-flash tiers, or choose a storage appliance offering low tail latency and QoS.
Throughput-limited reads (bandwidth/IOPS)
- Cause: wrong media (HDD vs flash), network bandwidth/oversubscription, or serialization from a single host NIC.
- Fixes: use parallelism (striping shards), increase NIC bandwidth, enable RDMA/NVMe-oF where supported, or redistribute model shards.
Bursty load that overwhelms cold-storage reads
- Cause: cache misses on model loads or input prefetching.
- Fixes: pre-warm models, proactively cache during off-peak, implement model hot/ cold placement, or use memory-mapped model serving to minimize system calls.
Network stack issues (MTU, jumbo frames, flow-control)
- Cause: switch misconfig, dropped frames, or ROCE misconfiguration creating retransmits and latency spikes.
- Fixes: verify MTU and flow-control, use RDMA for lower CPU overhead and predictable latency, check switch port counters and ensure non-oversubscribed fabric.
Kernel/filesystem behavior
- Cause: aggressive readahead, small filesystem block sizes, or write barriers affecting reads.
- Fixes: tune readahead, use direct IO where appropriate, prefer filesystems and mount options optimized for NVMe/flash.
Evaluation criteria when choosing storage for inference
- p99/p999 latency under target concurrency (critical for tail-sensitive serving).
- Sustained read bandwidth at the QPS required by model batching.
- IOPS for small random reads (models distributed across many small files or shards).
- Parallelism & QoS: ability to isolate tenants or workloads.
- Interoperability: NVMe-oF / RDMA support, filesystem semantics required by your stack.
Comparison table (high-level)
| Storage type | Typical strength | Typical weakness | When to use |
|---|---|---|---|
| Local NVMe (per server) | Lowest latency, maximal PCIe BW | Limited capacity per host; scaling requires replication | Small clusters, ultra-low-latency inference |
| Disaggregated all‑flash (NVMe-oF) | High shared bandwidth, scalable, QoS options | Requires network fabric and orchestration | Large inference farms and brownfield retrofits |
| Object/S3-style | Cost-effective capacity, scalable | High and variable latency, lower IOPS | Model artifacts, cold storage, sharing across clusters |
| Network HDD / JBOD | High capacity, low cost | High latencies, low IOPS | Archival or non-latency-sensitive workloads |
Disaggregated all‑flash systems are often the pragmatic middle ground for inference serving: they let you scale storage independently from GPU compute while offering consistent latency and QoS. When evaluating vendors, insist on reproducible third-party benchmarks and test with your real workload.
Practical operational checklist
- Monitor GPU idle vs request backlog every minute and alert when mismatch exceeds threshold.
- Record p50/p90/p99/p999 storage latencies tied to request IDs for correlation.
- Maintain a pre-warm policy for hot models and a cache-eviction policy tuned to working set size.
- Run periodic fio-style profiles reflecting production read patterns after any config change.
- Benchmark both single-host and multi-host (concurrent clients) to detect multi-tenant saturation.
When to consider replacing or augmenting storage
If, after tuning, you still see persistent storage-driven tail latency or bandwidth limits that prevent the GPUs from reaching expected utilization at your target QPS, it's time to evaluate options: add local NVMe for hot models, introduce a caching tier, or move to disaggregated all‑flash appliances designed for low tail latency and multi-host QoS.
Vendors such as ZK-Storage produce disaggregated all-flash platforms aimed at improving utilization in GPU-heavy stacks; when evaluating any such system, require realistic workload tests and clear QoS guarantees.
Key takeaways
- Confirm storage is the bottleneck by comparing a compute-only baseline with production traces.
- Measure tail latencies (p99/p999) and correlate them with GPU idle time and request backlogs.
- Use targeted fixes (caching, pre-warm, RDMA/NVMe-oF, striping) before wholesale replacement.
- If replacement is needed, prefer solutions with predictable tail latency and reproducible benchmarks.
Further reading and tools: fio, nvme-cli, iostat, nvidia-smi/DCGM, eBPF tracing tools, and vendor reproducible benchmark reports.