Skip to content

CONCEPT Cited by 2 sources

Circular dependency (deployment context)

A circular dependency in the deployment context is the failure mode where the act of deploying a fix for a service depends — directly or indirectly — on the very service the fix is trying to restore. If the service is down, the fix can't ship; if the fix can't ship, the service stays down; mean time to recovery expands unboundedly.

The canonical instance is dogfooded SaaS: GitHub deploys itself on github.com, Amazon sells on AWS, Cloudflare operates Cloudflare via Cloudflare Workers, Google depends on Google's auth. Dogfooding is a quality-and-speed win in steady state and a reliability anti-feature during incidents. GitHub frames the baseline mitigations as "maintaining a mirror of our code for fixing forward and built assets for rolling back" — but circular dependencies can still re-enter through deploy scripts themselves. (Source: sources/2026-04-16-github-ebpf-deployment-safety)

Three classes (GitHub taxonomy)

The GitHub Engineering post catalogues three classes, ordered by detectability:

  1. Direct dependency. The deploy script itself pulls an artifact from the impaired service. Example: a MySQL deploy script runs curl https://github.com/foo/bar/releases/.... Detectable by static source review of deploy scripts. Easiest to catch but "many dependencies aren't identified until an incident occurs" because review is manual and team-by-team.

  2. Hidden dependency. A tool already installed on the host contacts the impaired service at runtime (e.g. auto-update check on startup). The deploy script calls the tool, the tool calls out, the call hangs or fails, and the script inherits the failure. Detectable only by runtime audit — what does this tool actually contact? — not by reading the deploy script. Tool-inventory audits + network observation find these; source review does not.

  3. Transient dependency. The deploy script calls an internal service (e.g. a migrations service, a secrets service) which, within its own code path, pulls something from the impaired service. The failure propagates back. Detectable only by walking the full call graph including internal services. Adding a transient dependency is especially easy because the deploy script's own source code looks clean — the violation lives one hop away.

Why dogfooded architectures re-learn this repeatedly

  • Every team adds new scripts + new tools continuously. Today's clean deploy script may pick up a transient dependency tomorrow when a downstream service's team adopts a new library.
  • Tool updates are opaque. A pinned tool on the host silently picks up a new auto-update-check call in its next patch release.
  • Review is structurally incomplete. Static source review only catches class 1; audits at tool-update time only catch class 2; dependency-graph walks only catch class 3 if the graph is maintained.
  • Failure is rare in steady state — dogfooded internal services are reachable 99.9x % of the time, so a circular dependency is latent most of the year, surfacing only during the worst possible conditions (a real incident).

The tribal-knowledge shape doesn't scale. A structural fix — enforcement of the invariant "deploy scripts cannot talk back to the impaired service" — is what GitHub reaches for.

Sibling shape: forecast-context circular dependency

The same structural shape — output depends on input, input depends on output — reappears in the predictive-autoscaling control-loop context. MongoDB's 2026-04-07 retrospective names it "circular dependency" explicitly:

"We can't just train a model based on recent fluctuations of CPU, because that would create a circular dependency: if we predict a CPU spike and scale accordingly, we eliminate the spike, invalidating the forecast." (Source: sources/2026-04-07-mongodb-predictive-auto-scaling-an-experiment)

Same structural hazard, different layer of the stack:

Deployment context Forecast context
Trigger "Fix the service" "Predict the metric"
Dependency Fix depends on service being reachable Metric depends on control action
Symptom Can't ship during incident → MTTR expands unboundedly Forecast accuracy degrades under use
Remedy Mirror / independent infra + cGroup-scoped isolation Forecast exogenous inputs (concepts/customer-driven-metrics), then model the endogenous response

Both are named "circular dependency" in their respective sources. The forecast-context variant is documented in detail on the concepts/self-invalidating-forecast page; this page retains its focus on the deployment-context shape.

Structural fix: deploy-script-scoped egress filter

GitHub's post introduces eBPF + cGroups as the enforcement primitive:

  • Place deploy scripts in a dedicated cGroup.
  • Attach eBPF programs to that cGroup only.
  • Block outbound traffic to the dogfooded service from that cGroup only — the rest of the host (which may be serving customer traffic) is unaffected.
  • Crucially, this fails the deploy script visibly and immediately rather than silently hanging — surfacing the circular dependency as a deterministic pre-production test rather than a post-incident lesson.

See patterns/cgroup-scoped-egress-firewall and patterns/dns-proxy-for-hostname-filtering for the specific shape, and concepts/egress-sni-filtering for the adjacent hostname-filtering primitive in AWS Network Firewall (different tradeoffs: middlebox vs on-host, SNI vs DNS, VPC-wide vs cGroup-scoped).

  • concepts/blast-radius — scope-of-damage concept; a circular dependency widens blast radius by coupling recovery to the same scope as the outage.
  • concepts/control-plane-data-plane-separation — the structural separation that prevents control-path outages from taking out data-path traffic; circular-dependency failure modes often violate this across a dogfooding boundary.
  • concepts/grey-failure — sibling pathology (a service partially working, cascading failures); circular deps are structurally deterministic where grey failure is probabilistic.

Seen in

Last updated · 200 distilled / 1,178 read