Skip to content

SYSTEM

Sidekiq

Sidekiq is a Ruby background-job framework by Mike Perham (mperham/sidekiq) using Redis as the durable job store. Dominant async-job substrate for Ruby on Rails applications.

Architecture in one paragraph

Applications enqueue jobs by calling Worker.perform_async(args) (or .perform_in / .perform_at for scheduled). The job is serialised as a hash and pushed to a Redis list under a queue key. Sidekiq server processes pop from these lists, deserialise, and invoke Worker#perform in a thread pool. Failed jobs go through a retry queue with exponential backoff and ultimately a dead-letter set.

Middleware chains

Sidekiq exposes two middleware chains as composition points:

  • Client middleware — runs in the calling process at perform_async time, before the job is written to Redis. Receives (worker_class, job_hash, queue, redis_pool). A middleware that yields passes the job through; one that returns false / nil / falsy short-circuits the enqueue — nothing ends up in Redis. This is the hook feature-flagged job-enqueue rejection rides on.
  • Server middleware — runs in the worker process around perform. Wraps execution; can retry / skip / log / trace.

Both chains are configured in config/initializers/sidekiq.rb:

Sidekiq.configure_server do |config|
  config.client_middleware do |chain|
    chain.add(SidekiqMiddleware::SidekiqJobsFlipper)
  end
end

Enqueue APIs

  • Worker.perform_async(args) — enqueue immediately. One Redis round-trip per call.
  • Worker.perform_in(delay, args) / .perform_at(time, args) — enqueue with a fixed delay or absolute run-at.
  • Worker.perform_bulk([args1, args2, ...]) (Sidekiq 6+) — enqueue N jobs in a single Redis command. The amortisation lever for scheduler bulk-enqueue. PlanetScale uses batches of 1,000.
  • Worker.set(wait: duration).perform_later(args) (ActiveJob) — delayed enqueue, composable with jitter for jittered scheduling.

Sidekiq Enterprise unique jobs

Sidekiq Enterprise adds unique_for: + unique_until: worker options (concepts/sidekiq-unique-jobs):

sidekiq_options unique_for: 1.minute, unique_until: :start

Framework-level enqueue deduplication: a duplicate perform_async for the same (class, args) within the window is rejected silently. Complements application-level idempotence defences (state re-check, DB locks) as the third layer of defence against duplicate enqueues — particularly useful when paired-scheduler architectures deliberately produce duplicate enqueues.

Retry semantics

Default: 25 retries with exponential backoff, jitter, ultimately a dead-letter set. Retry happens on any unhandled exception. Combined with scheduler re-enqueue and user enqueue, this is why idempotent job design is load-bearing — the same logical work can arrive via three independent paths.

Seen in

  • — Canonical disclosure of Sidekiq uniqueness as a webhook-send dedup layer (2023-11-21, Mike Coutermarsh). PlanetScale runs webhook deliveries as Sidekiq jobs and reuses Sidekiq Enterprise unique_for — canonicalised earlier in the 2022-02-17 self-healing-jobs post as a duplicate-enqueue dedup for paired-scheduler architectures — to collapse rapid duplicate webhook enqueues into a single send. Verbatim: "Duplicate webhooks in quick succession get rejected, resulting in only a single unique webhook being sent out from our service, as well as limiting the number of webhooks we need to process." Composes into patterns/defense-in-depth-webhook-abuse-mitigation alongside API rate limits + isolated workers + send timeout + per-DB quota. Also canonicalises Sidekiq as the substrate hosting the isolated Kubernetes webhook worker tier (patterns/isolated-egress-proxy-for-user-urls) whose egress flows through Envoy for SSRF defence.
  • — canonical wiki introduction (2022-08-15, Elom Gomez). PlanetScale's application-tier Rails backend installs a client middleware that checks per-job-class Flipper feature flags and short-circuits enqueue if a disable_<classname> flag is set. Pattern: patterns/feature-flagged-job-enqueue-rejection.
  • — canonical self-healing-architecture disclosure (2022-02-17, Mike Coutermarsh). Same Rails backend architected around a paired scheduler-reconciler pattern: every job has a companion scheduler job that runs periodically and re-enqueues work from authoritative DB state, so flushing Redis is non-destructive. Introduces the wiki to Sidekiq's perform_bulk (batch-of-1,000 amortisation), perform_with_jitter (custom ApplicationJob helper for spreading bulk enqueue over a window), and Sidekiq Enterprise unique_for / unique_until for framework-level duplicate rejection. Same post-2022 PlanetScale application tier as the kill-switch post, sibling canonicalisation of the dedup stack.
Last updated · 542 distilled / 1,571 read