CONCEPT Cited by 1 source
Request queue depth metric¶
Request queue depth — the number of in-flight requests waiting inside a service — is a saturation leading indicator that frequently localises an upstream bottleneck when rate metrics (queries per second, requests per second) look normal.
Why it localises upstream bottlenecks¶
If a service's query rate is flat but its internal queue depth is growing, the bottleneck is downstream — the service can accept queries but can't clear them fast enough because something it depends on has slowed. This inverts the usual "traffic is up" root-cause hypothesis: the problem is latency-induced queueing, not volume.
Instrumentation examples¶
- DNS resolvers (Unbound): the
dump_requestlistcontrol command exposes the resolver's internal todo-list of queries awaiting upstream response. Depth growth = upstream is timing out. - Thread pools / worker queues: pending-work-queue length is the canonical form of this metric.
- Database connection pools: waiters-count on the pool.
- OS run queues: concepts/run-queue-latency is the kernel-scheduler-altitude sibling of this metric.
Seen in¶
- Stripe — The secret life of DNS packets (2024-12-12). Stripe's DNS query rate looked flat during hourly spikes, so "too many queries" was not the hypothesis. The request-list depth metric on Unbound was growing, which pointed the investigation at upstream (VPC resolver) response latency. Verbatim: "An increase in this metric indicates that Unbound is unable to process messages in a timely fashion, which may be caused by an increase in load. However, the metrics didn't show a significant increase in the number of DNS queries." Canonical instance of queue-depth as leading indicator when rate metrics don't move.