Skip to content

PATTERN Cited by 1 source

Queue-sharded migration

When a bulk migration exceeds per-invocation platform limits (e.g. subrequest caps), shard the work by logical key and fan it out via a durable queue with at-least-once delivery, ensuring no unit of work can silently drop.

Problem

A Worker has a fixed subrequest limit per invocation. A package with thousands of files exceeds this limit in a single pass. Parallelizing across Workers doesn't help — each still hits the same ceiling.

Solution

Shard the migration by a natural key (e.g. package name). For each shard, enqueue a message to a durable queue. A consumer processes each message independently, staying within per-invocation limits. The queue's at-least-once guarantee means no shard can silently fall out of the migration.

Trade-offs

  • Pro: no package silently drops; natural parallelism; composable with per-invocation retry
  • Con: at-least-once requires idempotent consumers; adds queue infrastructure; progress tracking needs a separate mechanism

Canonical instance

cdnjs KV → R2 migration hit the 1,000-subrequest limit. Work was sharded by package name and fanned out across Cloudflare Queues (Source: sources/2026-07-30-cloudflare-dogfooding-at-scale-migrating-cdnjs-to-cloudflares-developer-platform).

Seen in

Last updated · 604 distilled / 1,840 read