If there is one conversation I have had more than any other in my career, it is explaining to someone why “the storage has plenty of throughput” doesn’t mean “the application will feel fast.” Latency and throughput are both critical, but they measure different things, they are affected by different bottlenecks, and they matter differently depending on the workload. This article is my deep dive into how each one shapes real-world storage performance, and how I think about the relationship between them.
Defining the Two Metrics Clearly
Latency is the time it takes for a single I/O request to complete, from the moment it is issued to the moment the acknowledgment is received. It’s usually measured in milliseconds (ms) or microseconds (µs).
Throughput is the total volume of data moved per unit time, typically measured in MB/s or GB/s. It reflects aggregate capacity, not the experience of any single request.
A helpful way I explain this to people: imagine a highway. Latency is how long it takes one car to get from the on-ramp to the off-ramp. Throughput is how many cars pass a given point per hour. You can have a highway where every individual car travels quickly (low latency) but the total volume of cars is low (low throughput) if there are few lanes. Conversely, you can have massive volume (high throughput) but each car crawls (high latency) if the highway is congested.
The Relationship Between Latency, Throughput, and Queue Depth
These three are tied together by a version of Little’s Law, which I use constantly when reasoning about storage behavior:
Throughput = Queue Depth / Latency
Or rearranged:
Latency = Queue Depth / Throughput
This tells me something important: for a fixed device, increasing queue depth increases throughput — up to a point — but beyond that saturation point, latency increases sharply while throughput barely improves at all. This is the classic “knee of the curve” that I look for in every performance test.
| Queue Depth | Throughput (IOPS) | Avg Latency |
|---|---|---|
| 1 | 5,000 | 0.2 ms |
| 8 | 35,000 | 0.23 ms |
| 32 | 90,000 | 0.35 ms |
| 64 | 105,000 | 0.6 ms |
| 128 | 108,000 | 1.2 ms |
| 256 | 109,000 | 2.4 ms |
Notice how throughput plateaus around queue depth 64–128 while latency keeps climbing. Pushing queue depth past this point buys almost nothing in throughput but hurts every single request’s response time.
Why Latency Matters More for Some Workloads
Transactional and interactive workloads — OLTP databases, VDI, web application backends — are extremely latency-sensitive because they issue small, dependent I/O requests where each operation waits on the previous one to complete (think: a database commit that must flush its log before returning success to the application). In these cases, throughput headroom is almost irrelevant if latency is high; the application feels slow no matter how much theoretical bandwidth is available.
I’ve seen production database issues where the array reported only 20% utilized bandwidth, yet users complained about “slowness” — the real culprit was latency introduced by an overloaded storage controller CPU or a congested SAN fabric path, not a lack of throughput capacity.
Why Throughput Matters More for Other Workloads
Backup jobs, video streaming/rendering, big data analytics (Hadoop/Spark), and large sequential data transfers care primarily about throughput. These workloads issue large, often independent, sequential I/O requests where a bit of extra per-request latency is irrelevant as long as the aggregate data movement rate is high.
Example: Backing up 20 TB nightly with an 8-hour window
Required throughput = 20,000 GB / (8 × 3600 s) ≈ 0.7 GB/s minimum sustained
For this scenario, I care about sustained MB/s far more than microsecond-level latency per I/O.
Root Causes of High Latency
- Controller CPU saturation — inline deduplication/compression consuming cycles that would otherwise service I/O.
- Cache misses — forcing requests down to slower backend media.
- Network/fabric congestion — oversubscribed ISLs, packet retransmits, or insufficient network buffers on iSCSI/NFS.
- Queue depth mismatch — application driving far more concurrent requests than the storage path can service without backing up.
- RAID write penalty — parity-based RAID levels multiplying the actual I/O work per logical write.
- Thin provisioning metadata overhead — extra lookups needed to map logical to physical blocks, especially on heavily fragmented volumes.
- Noisy neighbors — shared storage pools where one workload’s I/O burst degrades everyone else’s latency.
Root Causes of Limited Throughput
- Interface/link speed ceiling — a 10GbE or 8Gb FC link simply cannot exceed its physical maximum regardless of how fast the backend media is.
- Insufficient parallelism — a workload using a single thread/queue can’t drive enough concurrent I/O to saturate a fast array.
- Small block sizes — throughput scales with block size at a given IOPS ceiling; tiny I/O sizes cap throughput even on very fast storage.
- Insufficient spindle/drive count — for HDD-based arrays, aggregate throughput scales with the number of drives in the stripe.
Measuring Both Together
I never look at latency or throughput in isolation — I always look at them together, ideally as a scatter or dual-axis chart across increasing load levels. Tools I use:
# fio - measure both latency and throughput under varying queue depths
fio --name=test --ioengine=libaio --rw=randrw --rwmixread=70 \
--bs=8k --iodepth=32 --numjobs=4 --size=2G --runtime=120 \
--time_based --group_reporting
# esxtop - watch DAVG (device latency) alongside throughput columns
esxtop
# then press 'u' for disk device view
# Cisco MDS - check FC port throughput and error counters together
show interface fc1/1 counters detailed
Diagram: Where Latency Accumulates Along the I/O Path
[Application] --(app queue latency)-->
[OS/Filesystem] --(scheduler latency)-->
[HBA/NIC Driver] --(queue depth wait)-->
[SAN Fabric/Network] --(propagation + congestion)-->
[Storage Controller] --(CPU + cache lookup)-->
[Physical Media] --(seek/access/program time)-->
Total Latency = sum of every hop above
I map this out explicitly during troubleshooting because it reminds me that “storage latency” as reported by the array is only one segment of the full end-to-end latency the application actually experiences.
Real-World Enterprise Example
I worked on a SQL Server cluster where transaction log latency crept from under 1ms to nearly 8ms over several months. Throughput on the array looked completely normal — nowhere near saturated. The root cause turned out to be a gradual increase in snapshot retention on the same volume, which increased metadata overhead on every write. Reducing snapshot frequency and moving the transaction log to a dedicated, non-deduplicated flash pool brought latency back under 1ms, with no throughput-related metric ever flagging the issue.
In another case involving a media production NAS, users complained about slow rendering pulls. Latency per request was fine (under 2ms), but aggregate throughput was capped because the NAS was still on a single 10GbE link shared by dozens of editors. Adding link aggregation (LACP) across four 10GbE ports and re-striping the workload across more spindles resolved it — a pure throughput-scaling problem, unrelated to latency.
Scalability: How Latency and Throughput Behave as Systems Grow
As I scale a storage system — adding drives, nodes, or hosts — latency and throughput don’t scale identically, and understanding why matters for capacity planning.
- Throughput scales roughly linearly with additional drives/nodes in most well-designed scale-out and RAID architectures, since more parallel media means more aggregate data movement capability.
- Latency does not automatically improve with scale — in fact, in poorly designed distributed systems, adding nodes can increase latency due to additional network hops, consensus/replication overhead, or metadata lookups across more nodes.
This is why I always test latency behavior specifically at scale, not just throughput, before committing to a scale-out architecture for a latency-sensitive workload. A system that throughput-scales beautifully to 20 nodes can still be a poor fit for a millisecond-sensitive OLTP database if per-request latency creeps upward with each added node.
Continuous Monitoring Practices
I treat latency and throughput monitoring as a continuous discipline rather than a point-in-time check:
- Percentile-based alerting: alert on p95/p99 latency crossing a threshold, not just averages, since averages mask the tail latency that actually drives user complaints.
- Throughput utilization trending: track sustained throughput as a percentage of link/device maximum over time to catch approaching saturation before it becomes an outage.
- Correlated dashboards: I build dashboards that plot latency and throughput on the same timeline so I can visually spot the “knee of the curve” moment when a workload transitions from healthy to saturated.
# Prometheus node_exporter query example (conceptual) for disk latency percentile
histogram_quantile(0.99, rate(node_disk_io_time_seconds_total[5m]))
Comparing On-Prem vs. Cloud Latency/Throughput Characteristics
| Environment | Typical Latency | Typical Throughput Scaling | Notes |
|---|---|---|---|
| On-prem all-flash SAN | 0.2–1 ms | Scales with controller/drive count | Full control over tuning, requires in-house expertise |
| On-prem HDD-based NAS | 5–15 ms | Scales with spindle count | Cost-effective for capacity, poor for latency-sensitive workloads |
| Cloud block storage (e.g., high-performance tier) | 0.5–3 ms | Scales with provisioned IOPS/throughput tier | Latency includes network hop to storage backend |
| Cloud object storage | 10–200+ ms | Very high aggregate throughput, poor single-request latency | Best for large sequential, not transactional workloads |
This comparison is why I never recommend moving a latency-sensitive transactional workload straight to object storage or a low-tier cloud disk without validating actual measured latency first — theoretical throughput numbers can look great on paper while masking a latency profile that’s completely wrong for the workload.
Common Mistakes
- Treating latency and throughput as interchangeable when sizing storage for a new application.
- Benchmarking only at low queue depth (missing throughput ceiling) or only at very high queue depth (missing real-world latency behavior).
- Ignoring percentile latency (p95/p99) in favor of averages, which hide the worst-case experience that users actually notice.
- Assuming that adding more throughput capacity (bigger pipe) will fix a problem that is actually rooted in latency (e.g., CPU-bound controller).
FAQs
Q: Can I have low latency and low throughput at the same time? Yes — a lightly loaded, well-tuned system with a narrow link (e.g., 1GbE) can have excellent per-request latency but a low overall throughput ceiling.
Q: Can I have high throughput and high latency at the same time? Yes — this is common under heavy queue depth: the pipe is full and moving a lot of data, but each individual request waits longer to complete.
Q: Which metric should cloud storage buyers focus on? Both, but check the specific SLA numbers for the tier you’re buying — cloud providers publish separate IOPS, throughput, and latency guarantees (e.g., AWS io2 Block Express, Azure Premium SSD v2) that don’t automatically scale together.
Q: Why does my application feel slow even when the array dashboard shows low latency? The array only reports latency from its own perspective — the time from when it receives a request to when it responds. It cannot see network transit time, HBA queuing, or application-level wait states upstream of it. I always measure latency as close to the application as possible, not just at the array, to get the true end-to-end picture.
Q: Does adding more cache always reduce latency? Only up to the point where the working set fits in cache. Once the active dataset exceeds cache capacity, adding more cache gives diminishing returns, and the fix shifts toward faster backend media or better data placement rather than more cache.
A Quick Mental Model I Use
Whenever I’m handed a new workload to size or troubleshoot, I ask myself two questions in order: “does this workload care about how fast one request finishes, or how much data moves overall?” and “at what queue depth does this workload actually operate in production?” Those two answers, more than any spec sheet number, tell me whether I should be optimizing for latency or throughput — and they keep me from over-engineering a solution for the wrong problem.
Summary
Latency and throughput describe two different dimensions of storage performance — how fast a single request completes versus how much data moves overall — and they are connected through queue depth via Little’s Law. Understanding which one matters for a given workload, and knowing where each accumulates along the I/O path, is essential for accurate sizing, effective troubleshooting, and realistic performance expectations.
References
- SNIA — Solid State Storage Performance Test Specification: https://www.snia.org
- VMware — esxtop Performance Metrics Guide: https://docs.vmware.com
- NetApp — Performance Latency and QoS Documentation: https://docs.netapp.com
- Dell EMC — PowerMax Latency and Throughput Sizing Guide: https://www.dell.com/support
- Cisco — Fibre Channel Performance Monitoring Guide: https://www.cisco.com/c/en/us/support
- AWS — EBS Volume Performance Documentation: https://docs.aws.amazon.com/ebs