Skip to content

DROPBOX 2026-03-25 Tier 2

Read original ↗

Dropbox — Reducing our monorepo size to improve developer velocity

Summary

Dropbox's server-side monorepo — a single Git repository holding most of the company's backend code across many services and libraries — had grown to 87 GB, making the initial git clone take >1 hour and repeatedly costing CI jobs that start from a fresh clone. The repo was gaining 20–60 MB/day (spikes >150 MB/day), putting it on course to hit the GitHub Enterprise Cloud 100 GB repo hard limit within months. The team traced the growth not to committed payload (binaries / deps / generated files were ruled out) but to a Git delta-compression heuristic: Git pairs similar files for delta storage using only the last 16 characters of the file path, and Dropbox's i18n layout i18n/metaserver/[language]/LC_MESSAGES/[filename].po put the language code earlier than that window, causing routine translation updates to produce deltas between different languages instead of within the same language — producing pathologically large pack files. A local repack with Git's experimental --path-walk flag shrank 84 GB → 20 GB, confirming the hypothesis, but --path-walk is incompatible with GitHub's server-side optimizations (bitmaps, delta islands) so could not be the production fix. GitHub instead ran a server-side repack with tuned --window=250 --depth=250 on a mirror (84 GB → 20 GB in ~9h), then rolled the same repack into the production repo one replica per day over a week. Final result: 87 GB → 20 GB (77% reduction), clone time >1h → <15 min. No fetch success / push success / API-latency regression. The team also re-shaped the i18n directory layout so delta pairing falls inside the 16-char window going forward, and stood up a recurring repo-stats dashboard (patterns/repo-health-monitoring) tracking size / growth rate / clone time / per-subtree distribution so the next regression is caught early.

Key takeaways

  • Repo-size growth isn't just about what you commit — it's about how Git decides to compress it. Standard explanations (large binaries / accidentally-committed deps / generated artefacts) were ruled out first; the actual cause was structural mismatch between the i18n directory layout and Git's 16-char delta-pairing heuristic. When your usage pattern diverges from a tool's internal assumption, "performance degrades quietly over time." (Source: sources/2026-03-25-dropbox-reducing-monorepo-size-developer-velocity)
  • Git's default delta heuristic pairs files using only the last 16 characters of the path. Dropbox's paths i18n/metaserver/[language]/LC_MESSAGES/[filename].po put the language code before the 16-char window, so Git was computing deltas across different languages instead of within the same language — producing oversized deltas from routine translation updates. See concepts/git-delta-compression.
  • The experimental --path-walk flag fixed it locally (84 → 20 GB in ~9h) but couldn't be the production fix. --path-walk walks the full directory structure for delta-candidate selection but is incompatible with GitHub's server-side optimizations (bitmaps, delta islands) used to keep clone/fetch fast. Local experiments proved the root cause; they couldn't ship.
  • You can't repack locally and push the improvement back to GitHub. GitHub rebuilds transfer packs dynamically on the server per client based on what that client is missing, so clone/fetch sizes are determined by the server's packing config, not by any mirror's. A permanent fix has to run on the server. (Source: sources/2026-03-25-dropbox-reducing-monorepo-size-developer-velocity)
  • Tuned server-side repack with --window=250 --depth=250 was the GitHub-supported alternative. These parameters widen how far Git searches for similar objects and how many layers of deltas it will stack; they take much longer (~9h on Dropbox's mirror) but stay compatible with all GitHub infrastructure. Result: 84 GB → 20 GB, matching the --path-walk result. See patterns/server-side-git-repack.
  • Validate a structural Git rewrite on a mirror first. Dropbox + GitHub test-repacked a mirror (78 GB → 18 GB), monitored fetch duration distribution, push success rate, and API latency, accepted "minor tail movement" as a tradeoff for a 4× size cut, confirmed no stability regressions — then scheduled production. See patterns/mirror-first-repack-validation.
  • Roll out the server-side repack gradually: one replica per day over a week, read-write replicas first, buffer time at the end for rollback. A repack changes how billions of objects are physically organised on disk for every clone/fetch/push of a central repository — same production-infra discipline as any other change affecting billions of objects in flight.
  • Treat repo health like production infrastructure — monitor continuously. Dropbox stood up a recurring stats job feeding an internal dashboard tracking overall repo size, growth rate, fresh clone time, and storage distribution across subtrees, so accelerating growth or creeping clone times are visible early rather than discovered via degraded engineer experience. See patterns/repo-health-monitoring.
  • Some fixes live at the boundary between your code and a managed platform. Dropbox could identify the root cause and validate the fix locally, but the actual permanent fix required a GitHub Support collaboration on a server-side-compatible approach. Strong platform relationships + shared debugging data (they brought growth-rate numbers, pack-size measurements, the --path-walk experiment result) turned into a concrete, supported remediation.

Key numbers

  • Repo size: 87 GB → 20 GB (77% reduction).
  • Initial clone time: >1 hour → <15 min in many cases.
  • Pre-fix growth rate: 20–60 MB/day typical, >150 MB/day spikes.
  • GHEC hard limit: 100 GB per repo — the approaching ceiling that made this an operational-risk issue, not just a velocity one.
  • Local git repack -adf --depth=250 --window=250 on a cloned mirror: ~9 hours wallclock (performance: 31205.079533000 s), 84 GB → 20 GB.
  • Local git clone --mirror (baseline measurement): ~46.6 min (performance: 2795.152366000 s), 84 GB on disk.
  • Test-mirror repack: 78 GB → 18 GB.
  • Git's default delta-pairing path window: 16 trailing characters.
  • Server-side repack rollout cadence: one replica per day, ~1 week total, read-write replicas first, rollback buffer at the end.

Architecture notes

Pre-fix state:

  • Dropbox server monorepo on GitHub Enterprise Cloud, ~87 GB, growing 20–60 MB/day.
  • Standard Git pack file layout on the server: git repack with default settings pairs delta candidates using only the last 16 characters of each path. Combined with i18n/metaserver/[language]/ LC_MESSAGES/[filename].po, this pairs .po files across languages rather than within a language.
  • Fresh-clone workflow: engineer/CI runs git clone → GitHub dynamically builds a transfer pack based on what the client lacks → client downloads → initial build/test loop proceeds.
  • CI: many jobs start from a fresh clone, so repo-size cost is paid repeatedly per CI run, not just once per engineer.

Investigation path:

  1. Rule out usual suspects: no large binaries, no leaked dependency trees, no generated-file commits accounting for the growth rate.
  2. Correlate growth spikes with translation commits → hypothesis that i18n .po traffic is producing outsized pack contributions.
  3. Read Git's repack internals; identify the 16-char trailing-path heuristic for delta candidate pairing (concepts/git-delta-compression).
  4. Local experiment on a git clone --mirror copy: git repack -adf --depth=250 --window=250 with experimental --path-walk → 84 GB → 20 GB in ~9h → root cause confirmed.
  5. Platform incompatibility: GitHub Support reports --path-walk incompatible with the server-side optimizations (bitmaps, delta islands) that GitHub relies on for fast clone/fetch; local fixes don't survive GitHub's dynamic pack construction on the server anyway (GitHub rebuilds transfer packs per request).

Fix path:

  1. GitHub recommends tuned --window=250 --depth=250 without --path-walk — more aggressive delta search within the default-heuristic pairing but stays compatible with server infrastructure.
  2. Test on a mirror of the repo on GitHub's side first (patterns/mirror-first-repack-validation): fetch latency distribution / push success rate / API latency monitored; 78 GB → 18 GB confirmed; "minor movement at fetch-latency tail" accepted as tradeoff.
  3. Gradual production rollout: one replica per day over ~1 week, read-write replicas first, buffer at the end for rollback. No regressions observed across fetch duration, push success, API latency.
  4. Structural follow-up: reshape i18n workflow so delta pairing falls within the 16-char window (re-layout .po file paths so that similar-language files are close enough in trailing characters for the default heuristic to pair them correctly).
  5. Operational follow-up: recurring stats job → internal dashboard tracking overall size, growth rate, fresh-clone time, per-subtree storage distribution (patterns/repo-health-monitoring).

Caveats

  • Vendor-supportive framing (Dropbox + GitHub collaboration retrospective); no disclosure of the specific --window=250 --depth=250 parameter-search cost beyond "higher values increase compute time during repacking."
  • No concrete numbers for CI-cost reduction — post describes CI improvements qualitatively ("spin up faster and run more reliably," internal services "less prone to timeouts").
  • No disclosure of monorepo content breakdown (which services, approximate per-subtree share, language mix); only i18n is named as the pathological case.
  • "Approaching GitHub's 100 GB hard limit" is named as operational risk but no specifics on what happens at the limit (soft degradation vs. hard reject).
  • No numbers on the post-fix growth rate or the resulting monthly headroom against the 100 GB limit.
  • The structural i18n workflow change is named but not specified — we know Dropbox "updated the i18n workflow to align with how Git's packing algorithm groups files," not what the new layout looks like.
  • Raw capture omits the CDN-served diagram image; the text in this post is the authoritative content.
Last updated · 200 distilled / 1,178 read