Skip to content

SYSTEM Cited by 1 source

Celery

Definition

Celery is a Python distributed task queue — a worker-pool + broker-based framework for running asynchronous and scheduled tasks outside the request/response path. A producer (typically a web app) enqueues a task to a broker (Redis, RabbitMQ, others); a pool of Celery workers consumes from the broker, executes the task, and (optionally) writes the result to a result backend so the producer can poll for completion.

Shape

┌───────────┐   enqueue   ┌────────┐   consume   ┌──────────┐
│ Producer  │────────────▶│ Broker │────────────▶│ Worker   │
│ (web app) │             │ (Redis,│             │ pool     │
└─────┬─────┘             │ Rabbit)│             └────┬─────┘
      │ poll / subscribe  └────────┘                  │ write
      ▼                                                ▼
┌───────────┐       result backend         ┌──────────────────┐
│ Producer  │◀──────────────────────────────│ Result backend   │
│ (web app) │  (task id → status + result) │ (Redis, DB, ...) │
└───────────┘                                └──────────────────┘

The broker and result backend are typically the same system (Redis is the canonical single-process default). Workers run in separate processes, can scale horizontally, and pick up tasks in FIFO-per-queue order by default with priority + routing available.

Load-bearing properties

  • Decouples latency from the HTTP request. Long-running synchronous work (LLM calls, IO fanout, batch processing) moves off the web dyno and onto the worker pool. The producer returns a task id immediately.
  • Horizontal scaling of workers. Adding workers scales throughput linearly up to the broker's fan-out ceiling.
  • Retry + backoff. Built-in retry semantics with exponential backoff; task idempotency is the caller's responsibility.
  • Scheduled tasks (Celery Beat). Cron-shape scheduling of periodic work, backed by the same worker pool.
  • At-least-once delivery by default. See concepts/at-least-once-delivery. Exactly-once requires caller-side dedup.

Seen in

  • Expedia STAR (2026-04-28) — STAR migrated from FastAPI's async/await + background tasks to Celery + Redis as it scaled up. Rationale verbatim: "As part of scaling up, we moved to Celery with Redis acting as the broker and result backend to store the state and results of tasks. This architecture aligns with STAR's request-response flow, and we don't need a streaming platform like Kafka, at least for now." Canonical wiki instance of Celery as the request-response async backend where a streaming platform (Kafka) would be over-engineering. (Source: sources/2026-04-28-expedia-expedias-service-telemetry-analyzer)

Comparison — Celery vs adjacent choices

Option Fit
FastAPI background tasks Good for sub-second fire-and-forget, no cross-process persistence; crashes lose work.
Celery + Redis/Rabbit Good for request-response async with durable task state, retries, and horizontal worker scaling. STAR's chosen point.
Kafka + consumer Good for unbounded streaming, at-least-once with consumer-group semantics, long retention. STAR's explicit non-goal.
Airflow / Temporal Good for multi-step durable workflows with explicit DAGs + retries + time semantics. Heavier than Celery.

The principle STAR's post demonstrates: match the queue to the traffic shape. Request-response with a few per-call tasks → Celery; streaming telemetry → Kafka.

Caveats

  • Stub page. Canonical Celery reference is the Expedia STAR ingest. Deeper Celery architecture (protocol, worker lifecycle, prefetch semantics, Beat internals) is not walked here.
Last updated · 433 distilled / 1,256 read