Skip to content

GITHUB 2026-10-21

Read original ↗

The bridge to agent-scale Git infrastructure

Summary

This is the second post in GitHub's agent-scale Git series (follow-up to Building Git infrastructure for agent-scale development), and it describes the immediate bridge between the old Spokes architecture and the compute–storage-separated end state. Over the prior year, while Git traffic grew exponentially, 99.99999% of repositories were migrated so they live in two places at once (GitHub's own data centers + Azure); the first copy of a repo landed in Azure's Central US region in August 2025, and today most repos have two of their five replicas in Azure, with Azure serving ~half of all Git traffic. The post makes two architectural moves. First, it changes the scope of the Spokes quorum from the whole repository to individual references — because Git's consistency requirements apply per-reference, and most concurrent writes touch different refs (agents/devs on their own branches). This lifts the repository-wide write lock that turned the quorum into a single one-in-flight queue, and gets the last ~0.00001% of (busiest) repos into Azure despite the ~25 ms cross-region round trip. Second, it makes object storage (Azure Blob Storage) a member of the quorum as a "virtual replica" — a fully ephemeral compute node that streams writes to the blob store and serves no reads — which then lets GitHub spin up stateless virtual read replicas on demand to scale reads independently. (Source: sources/2026-10-21-github-the-bridge-to-agent-scale-git-infrastructure)

Why now: the migration and the 25 ms wall

  • Driven by the agentic shift: exponential Git growth against a hard power ceiling on GitHub's data centers and their geographic region, plus physical supply-chain constraints. There was enough remaining capacity to migrate safely but not slowly.
  • Changing the storage architecture and the hosting platform simultaneously, under that growth, was judged too risky for a Tier 0 service. So the work was sequenced: move the data first, evolve the architecture once it arrived.
  • During migration, traffic follows the data: Spokes uses preferential routing to send each request to a replica in the same region/AZ. As replicas move to Azure, the Azure-servable share grows replica-by-replica until the last replica arrives and the old region stops receiving requests entirely.
  • The sticking point for the busiest repos is a ~25 ms round trip: once a replica lives in the target Azure region it is ~1,000 miles from the data centers, and a write requires multiple round trips. A repo already near its throughput ceiling can't absorb that on every serialized write — it would fall further behind with every push.

Key takeaways

  1. The old quorum waits for every healthy replica, so the slowest one sets write latency AND throughput. Spokes' quorum only needs a majority to commit, but it waits for every healthy replica to respond (up to a timeout) before moving on — so slow replicas delay every write rather than falling behind. Because each write holds a repository-wide lock until it commits, the slowest healthy replica sets both the latency and the throughput ceiling of the whole repo. (Source: sources/2026-10-21-github-the-bridge-to-agent-scale-git-infrastructure)

  2. Repository-wide locking is head-of-line blocking. Holding a repo-wide lock for the entire reference update (round trips included) "turns the quorum into a single, unordered request queue with one write in flight at a time." As the write rate grows, the queue grows and each write waits longer than the one before — classic head-of-line blocking.

  3. Per-reference serialization is the fix: don't serialize what doesn't need to be. Git's consistency requirements apply to each reference individually — once the server accepts a ref update, no client may see an older value, only that one or newer (true for fast-forward, force-push, or delete). But updates to different refs usually don't depend on each other. So GitHub narrows the quorum's scope from the whole repository to the individual reference. Only writes that contend on a shared ref must wait for each other; writes to different refs (agents/devs pushing their own branches) proceed concurrently, each still paying the cross-region latency but no longer queued behind one another. (Source: sources/2026-10-21-github-the-bridge-to-agent-scale-git-infrastructure)

  4. Measured result: 3–4× higher write throughput at lower latency. Per-ref serialization raises a repo's write ceiling "roughly in proportion to how many references are being written at once." In internal testing with realistic traffic, writes to different references "consistently saw 3–4× higher throughput at lower latency," and this is what gets the last remaining repos into Azure.

  5. Reads are 100–1,000× writes but compete with them today. Most repos are heavily read-skewed (100–1,000 reads per write), yet reads and writes hit the same replicas and compete for the same resources, so they can't be scaled independently. Reads would benefit from far more in-memory cache than they get today, but that memory is also needed for compaction — separating the two lets each get its own configuration. (Source: sources/2026-10-21-github-the-bridge-to-agent-scale-git-infrastructure)

  6. Object storage becomes a quorum member — the "virtual replica." The first step toward compute–storage separation is making Azure Blob Storage a special quorum member. The compute that represents it is GitHub's first fully ephemeral node: it bridges to object storage, streams incoming writes, and serves no reads. Because it's in the quorum, a write isn't acknowledged until it is durable in object storage.

  7. Per-reference serialization keeps the extra durability hop from becoming a bottleneck — and the virtual write replica coalesces I/O. Writing to object storage adds latency to each write, but — like the cross-region round trip — per-ref serialization keeps it off the throughput critical path. The virtual write replica also coalesces transactions on disjoint references into a single set of network I/Os to object storage, so a busy repo "pays for one round trip across many concurrent writes instead of one per write." For now the virtual write replica also handles compaction, and read replicas inherit the compacted state.

  8. Virtual read replicas scale reads on demand without joining the quorum. With the virtual write replica in place, GitHub can spin up as many virtual read replicas as a repo needs. Each warms its cache from Azure Blob Storage and starts taking read traffic without joining the quorum, using object storage as the source of truth and reconciling its local state against it in the background. In load testing, the current architecture "routinely found ceilings"; against virtual read replicas, the harness "hit its own limits first." (caching proxy tier over a durable store.)

  9. Two acknowledged limits of this bridge step. (a) Background reconciliation makes virtual read replicas a good fit for some read workloads but not all; serving every kind of read requires moving reconciliation in-line — fetching only the reference state and data changed since the replica last caught up. (b) Even after shifting to per-reference serialization, the quorum remains the bounding factor for pushing write throughput higher. (Source: sources/2026-10-21-github-the-bridge-to-agent-scale-git-infrastructure)

  10. Spokes is far more than a quorum — and the commit protocol is the part that must change. Spokes is also an intelligent routing/load-balancing layer, a unified API serving well over 100 billion operations per day, and a set of background "sweepers" for repository placement, repair, and rebalancing. Most of that applies equally to the new architecture; but GitHub concedes "the commit protocol that has taken GitHub to a billion repositories won't take it to the next billion." The remaining hard problem: Git objects are immutable and content-addressed (so they can be stored without anyone agreeing on them), and what's left to order is small — the challenge is ordering it as fast as possible without moving the bottleneck, while handling the second-order effects (where durable reference state lives, how compaction/GC scale). A future post (GitHub Universe) covers the end state and "why there's still a (small) consensus group somewhere in the stack."

Systems / concepts / patterns extracted

Operational numbers

Metric Value Notes
Repos living in two places (DC + Azure) 99.99999% "the rest are among the busiest"
First repo copy in Azure Aug 2025 Azure Central US region
Azure replica share today 2 of 5 replicas most repos
Git traffic served from Azure today ~half
Default replica count 5 balances durability/availability/scale
Cross-region distance / RTT ~1,000 miles / ~25 ms DC ↔ Azure target region
Read:write ratio 100–1,000 : 1 most repos are read-skewed
Per-reference write-throughput gain 3–4× internal testing, realistic traffic, lower latency
Spokes unified-API load >100 billion operations/day
Remaining busiest repos ~0.00001% the last to migrate

Caveats

  • The 3–4× write-throughput figure is from internal testing with realistic traffic, not disclosed production numbers; the post does not define the benchmark shape.
  • This is a bridge post: the virtual read replica's background reconciliation is explicitly not suitable for all read workloads yet (in-line reconciliation is deferred), and the quorum "remains the bounding factor" for write throughput — the end-state write path is deferred to the GitHub Universe post.
  • The post states object storage joins the quorum so writes aren't acked until durable there, but does not quantify the added per-write latency.
  • The predecessor post left open "how is strong-consistency visibility preserved once authoritative data is in object storage"; this post answers part of it (blob store as a durability-gating quorum member + per-ref ordering) but still defers the final write-ordering mechanism.

Source

Last updated · 854 distilled / 2,448 read