Skip to content

PATTERN Cited by 1 source

Serverless driver-worker pattern

Decouple image / event processing into a driver that orchestrates work distribution + workers that process each unit concurrently, each on its own scale-to-zero serverless substrate. Workers are stateless, one-worker-fault can't take the pipeline down, and each work-type gets an independent capacity + release cadence.

Shape

  • Driver: receives external events (object-created notifications, scheduled triggers, webhook arrivals); fans out to per-work-type queues. Typical substrate: Lambda fronted by SNS or by an EventBridge rule.
  • Per-work-type queue: SQS queue (with a dedicated DLQ) per downstream processing semantic. Enables independent backpressure, per-queue capacity raise-limits, and per-queue retention / visibility-timeout tuning.
  • Worker pool: Lambda consumers or container workers per queue; each invokes the correct backend (ML endpoint, DB mutation, external API). Workers are stateless — processing state lives in DynamoDB / external stores.

Why it pays

  • Independent scaling. Each worker type scales to its own traffic without coupling to sibling work types. A burst on one use case does not drain capacity from another.
  • Fault isolation. A buggy worker or a failing downstream endpoint hits its own queue's DLQ; other workers keep processing.
  • Per-component release cadence. Workers can deploy / roll back independently; the driver stays unchanged.
  • Filter-to-downstream. Workers that run expensive downstream state mutations (DB writes, notifications) only trigger on the minority of inputs that qualify; the driver never touches downstream state. Canonical framing in the AWS safety-monitoring source: "the inference layer only surfaces images where safety issues have been detected. This filtering is essential because it prevents components interacting with Amazon Aurora PostgreSQL from being overwhelmed by the raw volume of image data from hundreds of sites."

Required machinery

  • Per-queue DLQ so failed messages can be inspected / re-driven.
  • Per-queue alarm on depth + age, not just worker error rate.
  • Quota management — raising Lambda concurrent-execution limits per account when adding a new worker type is part of the rollout, not an afterthought.
  • Stateless-worker discipline — processing state belongs in an external store so workers can be replaced without coordination.

Seen in

  • sources/2026-04-01-aws-automate-safety-monitoring-with-computer-vision-and-generative-ai — canonical wiki instance. Each computer-vision use case (PPE detection, Housekeeping) has its own SNS → SQS → Lambda → SageMaker endpoint worker chain. "Each use case processes independently through its own queue, which invokes a SageMaker AI Endpoint hosting a computer vision model tailored to that scenario." "This pattern decouples image processing tasks, enabling independent scaling of different components while providing fault isolation. If one worker fails, it doesn't impact the entire pipeline." Failures per use case land in use-case-specific DLQs.
Last updated · 200 distilled / 1,178 read