Skip to content

CONCEPT Cited by 1 source

Race condition

A race condition occurs when a system's correctness depends on the relative timing or interleaving of operations, and at least one possible ordering produces incorrect behavior. Race conditions are among the hardest bugs to diagnose because they are non-deterministic: the same inputs produce different outcomes depending on scheduling, load, and environmental factors.

Characteristics in Production

Race conditions in production systems typically share these properties:

  • Intermittent failure — only triggered under specific timing windows.
  • Load-dependent — more likely under high concurrency or with larger payloads that take longer to process.
  • Heisenbug tendency — adding instrumentation changes timing enough to mask the bug.
  • Invisible to application-level observability — tracing and logging often report success because the application believes it completed its work.

The hyper Premature Shutdown Race

The canonical example in this wiki is the systems/hyper race condition where: - A Poll::Pending from poll_flush was discarded via let _ =. - Under backpressure (socket buffer full), the dispatch loop concluded the connection was done. - shutdown(fd, SHUT_WR) was called with megabytes still in the write buffer.

The bug existed across multiple major hyper versions for years but only surfaced when Cloudflare's December 2025 rearchitecture introduced a reader that paused for milliseconds longer than the previous one (Source: sources/2026-06-22-cloudflare-how-we-found-a-bug-in-the-hyper-http-library).

Debugging Approaches

  • Kernel-level tracing (strace, perf, eBPF) — records actual syscalls rather than what the application thinks happened. Critical for races in I/O paths.
  • Deterministic test harnesses — simulating the exact environmental condition (e.g., a mock socket that blocks after N bytes) to reproduce timing-sensitive bugs reliably.
  • Controlled reproduction environments — batch scripting that fires enough requests to hit the timing window statistically.

Seen In

Last updated · 559 distilled / 1,651 read