Skip to content

SYSTEM Cited by 4 sources

AWS SQS

Amazon SQS (Simple Queue Service) is AWS's managed message queue: durable, at-least-once delivery, essentially unlimited scale, with standard and FIFO variants. In system-design terms, it's the generic durable work-dispatch primitive between producers and consumers that can operate at different rates and need the queue to survive either side crashing.

Typical role in data pipelines

SQS commonly appears as the durability layer in warehouse-unload bridges: the warehouse exports into S3, an S3-event emits an SQS message, and the ingester consumes the queue and writes to the serving store. If the ingester crashes or the serving store throttles, the message stays in the queue; when the ingester comes back up, it resumes without data loss.

Canva uses exactly this shape in the counting pipeline: Snowflake unload → S3 → SQS → rate-limited ingester → service RDS. They call out SQS's durability specifically as the reason exported data "doesn't get lost". (Source: sources/2024-04-29-canva-scaling-to-count-billions; patterns/warehouse-unload-bridge)

Operational notes

  • At-least-once means consumers must be idempotent (outer-join upserts are one way to get there — see patterns/end-to-end-recompute).
  • Visibility timeout has to be tuned against downstream write latency; too low and messages double-deliver, too high and failed messages sit idle.
  • DLQs for poison-pill messages are a must in any serious pipeline.

Seen in

Last updated · 542 distilled / 1,571 read