Skip to content

PATTERN Cited by 1 source

Rate-limit-aware dequeue throttling

Problem

LLM and vision model endpoints enforce two distinct quotas: a requests-per-second (RPS) cap and a tokens-per-minute (TPM) cap. A naive task queue that dispatches work as fast as workers can claim it overwhelms downstream endpoints, triggers throttling, and leads to repeated retries — wasting both compute and quota.

Solution

Enforce throttling at dequeue time within the same transaction as FOR UPDATE SKIP LOCKED. Three configurable modes, selected per agent:

1. Concurrency cap

A MAX_CONCURRENT_TASKS parameter bounds in-flight tasks. Before dequeuing, count PROCESSING rows in the tasks table:

SELECT COUNT(*) FROM tasks WHERE status = 'processing';

If at or above the cap, no new task is dequeued. Anchoring on database row count (not local queue size) keeps the cap accurate across worker restarts, lease recoveries, and multi-replica deployments.

2. Token budget

A MAX_TPM parameter bounds projected token rate. The orchestrator estimates a task's token count, sums projected tokens across all PROCESSING tasks, and dequeues only if the sum plus the new task fits within budget.

3. Combined cap

When both are configured, the tighter constraint wins. This handles workloads that are concurrency-bound under one regime (many short tasks) and token-bound under another (single long document saturating per-minute quota).

Key properties

  • Transactional — throttle decision and row lock happen in the same transaction; no race between checking capacity and claiming work.
  • Stateless across replicas — the database is the single source of truth for in-flight work; no inter-replica coordination needed.
  • Adaptive — tasks that don't fit stay in the queue for the next dequeue cycle; no separate waiting queue or scheduling state.
  • No external rate limiter — no Redis-based token bucket or sidecar rate limiter needed.

Trade-offs

  • Token budget mode depends on estimated token counts which may be inaccurate for variable-length documents.
  • Counting PROCESSING rows on every dequeue adds a sequential scan (or index scan) to the hot path — acceptable for moderate throughput but may need a materialized counter at very high scale.
  • The combined mode's "tighter constraint wins" is simple but doesn't optimize for throughput in mixed regimes (e.g., could batch small tasks more aggressively when token headroom exists).

Seen in

Last updated · 591 distilled / 1,796 read