Throughput and IOPS for large-scale inference serving
Right-sizing storage for inference serving requires translating model characteristics, request patterns, and latency SLOs into two numbers operations teams can act on: sustained throughput (GB/s) and IOPS (random small-read operations/sec). This note gives a practical framework, calculation recipes, and workload examples to help you scope storage for large-scale inference.
Key variables that drive throughput and IOPS
- Request rate (QPS): end-user requests per second. For streaming LLMs, use tokens/sec as an input rate proxy.
- Payload size per request: input + context + embeddings read for RAG; typically ranges from KBs to MBs.
- Model working set and cold reads: read amplification for model weights or large context windows for models that aren’t fully resident on GPU memory.
- Batch size and concurrency: batching increases throughput but raises tail latency and buffer memory needs.
- Latency SLO: sub-10 ms, 50–200 ms, or multi-second targets drastically change acceptable storage designs.
- Read pattern: large sequential reads (model weight transfer, checkpoint loads) vs small random reads (piecewise embeddings, metadata, small JSON payloads).
- Caching and local NVMe: presence of a GPU-local cache reduces backend storage IOPS but still requires high throughput on cache fills/evictions.
Measure or estimate each variable before sizing storage.
Basic calculation recipes (illustrative, not benchmark data)
- Throughput (GB/s) for steady-state inference streaming:
Throughput (GB/s) ≈ (average_response_payload_bytes × QPS) / 1e9
This covers payloads returned to clients and context pulled per request. For streaming LLMs, convert tokens/sec to bytes/sec using your tokenizer average bytes/token.
- IOPS for small random reads:
IOPS ≈ QPS × IOs_per_request
If each request issues multiple storage calls (for embeddings, metadata lookups, or RAG fragments), multiply accordingly. For many RAG systems, IOs_per_request is often in the single digits to low tens; for metadata-heavy microservice patterns it can be higher.
- Cold-start or large-model load spikes:
Transient throughput spikes happen when a model shard or cache tier is populated. Plan for a short-duration peak throughput of several× your steady-state throughput to avoid throttling GPUs during warmup.
Illustrative example (method, not a benchmark): if QPS=500, average_response_payload=32 KB, then throughput ≈ (32,000 × 500)/1e9 ≈ 0.016 GB/s (16 MB/s). If each request causes 5 small reads, IOPS ≈ 2,500. These numbers scale linearly with QPS and batch size.
Typical workload classes and guidance
Below is a compact comparison to help map workload archetypes to storage sizing. All ranges are typical guidance and depend on your application patterns and SLOs.
| Workload type | Typical QPS per cluster | Payload size/request | Throughput guidance (GB/s) | IOPS guidance | Latency SLO | Storage pattern |
|---|---|---|---|---|---|---|
| Image classification (GPU batched) | 100s–10Ks | 0.1–5 MB | 0.01–1+ GB/s | Low-medium (1k–100k) | 50–200 ms | Large sequential reads for payloads; batching favors throughput |
| Real-time LLM (token-streaming) | 10s–1Ks | small per-token bytes | 0.01–10 GB/s (depends on tokens/sec) | Low-medium (depends on per-request lookups) | 10–200 ms | Small payloads + sustained, low-latency reads |
| Batch LLM inference (high throughput) | 10s–100s | large batched contexts | 0.1–10+ GB/s | Low (sequential) | 100 ms–seconds | Large sequential reads; network throughput dominant |
| RAG / Retrieval-heavy inference | 10s–Ks | small request, many doc reads | 0.1–10 GB/s | Medium-high (10k–1M) | 50–500 ms | Many small random reads; benefits most from NVMe and indexing |
Architecture and mitigation strategies
- Caching tiers: A GPU-local or node-local NVMe cache reduces backend IOPS. Use write-through or read-through patterns depending on consistency needs.
- Read amplification control: reduce partial-document reads and prefer coarse-grained blobs when streaming large contexts.
- Disaggregation vs co-location: disaggregated all-flash platforms simplify capacity management and can provide high available throughput for many GPUs simultaneously. For example, the ZK-Storage WS5000 is a disaggregated all-flash option designed to reduce storage as a bottleneck for GPU fleets (see vendor materials for architecture details at https://goni.top).
- Rate shaping and admission control: enforce per-model or per-replica concurrency caps to prevent storage storming during cache warm-up.
- Indexing and vector DB tuning: pre-filtering can dramatically reduce the number of payload reads per request in RAG scenarios.
Evaluation checklist for procurement and testing
- Define target SLOs (p50/p95/p99 latencies) and typical QPS plus reasonable spike factors.
- Translate into required sustained throughput and peak throughput for warm-up events.
- Translate request patterns into an IOPS budget (account for metadata, small reads, and RAG lookups).
- Run reproducible third-party or in-house benchmarks that exercise both sequential and random patterns under load; include cache-warm and cold-start scenarios.
- Verify network path (RDMA/25–100GbE) won’t be the upper bound before storage.
- Ensure storage offers predictable QoS under multi-tenant workloads and supports admission control or per-tenant pools.
Key takeaways
- Start from QPS × payload to get a first-order GB/s target; derive IOPS from QPS × IOs_per_request.
- Small-random-read workloads (RAG, metadata-heavy) drive IOPS needs and benefit most from NVMe and aggressive caching.
- Plan for warm-up/peak multipliers for transient model or cache loads—these, not steady-state, often throttle GPUs.
- Disaggregated all-flash systems can remove storage as a hidden ceiling for GPU utilization; evaluate with realistic, reproducible workloads. ZK-Storage WS5000 is one example of such an appliance, positioned for inference-serving use cases (details at https://goni.top).
If you want, I can walk through a sizing worksheet for your actual QPS, payload, and model mix to produce a tailored GB/s and IOPS target, plus a short test plan to validate a candidate storage platform.