Databricks

Databricks Engineering blog. Tier-3 source on the sysdesign-wiki: most posts are product/marketing/ML-methodology oriented and get skipped, but infra-architecture posts (Kubernetes, service mesh, data-platform internals) are worth ingesting when they appear.

Internally Databricks runs hundreds of stateless gRPC services per Kubernetes cluster across thousands of clusters in multiple regions, predominantly in Scala on a monorepo with fast CI/CD. That monoculture is the architectural enabler for several of their infra-platform choices — notably the proxyless service-mesh design.

Key systems

• systems/dicer — Databricks' open-sourced (2026-01) auto-sharder. Dynamic slice-range sharding with hot-key isolation + replication, eventually-consistent Assignments, state transfer across reshards. Used by Unity Catalog, Softstore, the SQL query orchestration engine, and "every major Databricks product".
• systems/unity-catalog — unified governance service; Dicer-backed sharded in-memory cache drove 90–95% hit rate and drastic DB-load reduction. Also the hub of customer-facing data meshes — federates external Iceberg catalogs and exchanges data via systems/delta-sharing.
• systems/delta-sharing — open cross-cloud / cross-metastore / cross-partner data-exchange protocol. Used by Mercedes-Benz for three deployment shapes (cross-hyperscaler, cross-region, external partner) on one wire protocol.
• systems/delta-lake — Databricks' open table format; Deep Clone is the incremental-replication primitive behind patterns/cross-cloud-replica-cache.
• systems/softstore — distributed KV cache built on Dicer; canonical example of Dicer's state-transfer (~85% hit rate preserved through rolling restarts vs. ~30% drop without).
• systems/databricks-endpoint-discovery-service — custom xDS control plane watching Kubernetes services/EndpointSlices, feeding both Armeria RPC clients (internal) and Envoy ingress (external) off one source of truth.
• systems/armeria — shared Scala RPC framework; host of embedded client-side LB + xDS subscription code.
• systems/storex — internal AI-agent platform for database debugging across the global fleet; central-first sharded architecture
• DsPy-inspired tool framework + snapshot-replay validation with judge LLMs.
• systems/dspy — Databricks-sponsored programmatic prompt framework; cited as inspiration for Storex's tool/prompt decoupling.
• systems/mlflow — Databricks-originated ML lifecycle platform; hosts Storex's judges primitive and prompt-optimization tooling.
• systems/lakebase — Databricks' serverless Postgres (Neon lineage, 2025 acquisition); Pageserver + Safekeeper durable storage, ephemeral Postgres compute VMs. 2026-04-20 CMK rollout ingested.
• systems/pageserver-safekeeper — the Neon-lineage page + WAL durable storage tier Lakebase inherits.
• systems/aws-kms / systems/azure-key-vault / systems/google-cloud-kms — the three cloud KMSes Lakebase's Customer-Managed Keys feature integrates with.
• systems/unity-ai-gateway — productised AI-gateway for coding agents + MCP governance (launched 2026-04-17). Three pillars: centralised audit in Unity Catalog, single-bill cost control via Foundation Model API + BYO external capacity, OpenTelemetry → UC-Delta-table observability. Clients ready at launch: Cursor, Codex CLI, Gemini CLI, with Claude Code via MLflow 3 tracing.
• systems/databricks-foundation-model-api — first-party inference for OpenAI/Anthropic/Gemini/Qwen underneath Unity AI Gateway; BYO external capacity supported.

Key patterns / concepts

• patterns/shard-replication-for-hot-keys — isolate a hot key into its own slice, replicate that slice across N pods (Dicer's answer to concepts/hot-key).
• patterns/state-transfer-on-reshard — migrate per-key state across pods during assignment changes so caches survive rolling restarts (Softstore).
• concepts/dynamic-sharding — continuously-adjusted Assignment driven by health + load signals; Dicer's core primitive.
• concepts/static-sharding / concepts/hot-key / concepts/split-brain — the three structural failure modes Dicer was built to replace.
• concepts/eventual-consistency — Dicer's Assignment consistency model; trade-off vs. Slicer / Centrifuge leases.
• concepts/soft-leader-election — key-affinity-as-coordinator, one of Dicer's named use cases.
• patterns/proxyless-service-mesh — mesh capabilities (discovery, L7 LB, health-aware routing, zone-affinity) via shared library instead of sidecars. Rejected Istio / Ambient Mesh explicitly.
• patterns/power-of-two-choices — the default LB algorithm embedded in Armeria clients.
• patterns/zone-affinity-routing — with capacity/health-driven spillover.
• patterns/slow-start-ramp-up — introduced after client-side LB surfaced cold-start issues on fresh pods.
• patterns/tool-decoupled-agent-framework — DsPy-inspired; tools-as-functions with docstrings, prompts and tools swap independently.
• patterns/snapshot-replay-agent-evaluation — production-state snapshots replayed through candidate agent configs, scored by a judge LLM — Storex's regression harness.
• patterns/specialized-agent-decomposition — per-domain agents (DB, traffic, …) collaborating on root-cause analysis.
• patterns/hackathon-to-platform — 2-day prototype → user-feedback iterations → platform; Storex's on-ramp.
• concepts/client-side-load-balancing — overall architectural posture for internal RPC.
• concepts/layer-7-load-balancing — why they left kube-proxy.
• concepts/xds-protocol — the control-plane-to-data-plane contract, used beyond sidecar meshes here.
• concepts/control-plane-data-plane-separation — EDS vs. client/Envoy.
• concepts/tail-latency-at-scale — the observed symptom that motivated the proxyless redesign.
• concepts/central-first-sharded-architecture — Storex's foundation: global coordinator + regional shards for data-residency + one auth model across 3 clouds / hundreds of regions / 8 regulatory domains.
• concepts/llm-as-judge — scoring primitive inside Storex's validation framework.
• concepts/data-mesh — Unity Catalog + Delta Sharing position as the Databricks-opinion answer to the mesh shape, with domain-owned data products + central governance + open exchange protocol.
• concepts/hub-and-spoke-governance — the UC governance posture; central catalog + federated spoke data, one policy surface across clouds/regions.
• concepts/cross-cloud-architecture / concepts/egress-cost — the forcing functions behind the Mercedes-Benz mesh design.
• patterns/cross-cloud-replica-cache — Delta Sharing + Delta Deep Clone as the canonical shape for bulk cross-cloud consumers.
• patterns/chargeback-cost-attribution — bytes-at-the-sync-tier → producer-side billing dashboard; the governance hygiene layer on top of the replica-cache pattern.
• concepts/envelope-encryption — the three-level CMK → KEK → DEK hierarchy the Lakebase CMK rollout articulates cleanly.
• concepts/cmk-customer-managed-keys — customer-held root of trust model; Lakebase realises it across both its storage and compute tiers.
• concepts/cryptographic-shredding — Lakebase revocation semantics: unwrap fails → data cryptographically inaccessible; Manager terminates compute VMs.
• patterns/per-boot-ephemeral-key — Postgres compute VMs generate a per-boot ephemeral key that dies with the instance; pairs with concepts/stateless-compute scale-to-zero.
• concepts/compute-storage-separation — Lakebase's Pageserver+ Safekeeper vs ephemeral Postgres compute split is a canonical OLTP-shape instance (cf. Aurora DSQL, Snowflake).
• concepts/coding-agent-sprawl — named problem class: engineering orgs simultaneously running Cursor + Codex + Claude Code + Gemini CLI + … ; Databricks itself is the stated example.
• concepts/centralized-ai-governance — three-pillar framing (security/audit + single bill + Lakehouse observability) that Unity AI Gateway instances, paralleling Cloudflare's internal-stack shape with different substrates.
• patterns/ai-gateway-provider-abstraction — Unity AI Gateway is the Databricks instance, specialised for coding-agent + MCP governance.
• patterns/central-proxy-choke-point — architectural posture: all coding-tool traffic funnels through one gateway, no second path to providers.
• patterns/unified-billing-across-providers — Foundation Model API as first-party default + BYO external capacity = one bill + per-developer (not per-tool) budgets.
• patterns/telemetry-to-lakehouse — coding-tool OpenTelemetry → UC-managed Delta tables; joinable with Workday / PR-velocity / capacity-planning data.

Recent articles

• 2026-04-17 — sources/2026-04-17-databricks-governing-coding-agent-sprawl-with-unity-ai-gateway (Launch post for Unity AI Gateway coding-agent support. Names coding- agent sprawl — engineers routinely mix Cursor + Codex + Claude Code + Gemini CLI + others in parallel — as the forcing function. Three-pillar answer: (1) centralised security + audit via Unity Catalog + MLflow tracing + single-SSO across all coding tools + Databricks-managed MCP servers; (2) single bill + cost controls via Foundation Model API first-party inference + BYO external capacity + per-developer (not per-tool) budgets — patterns/unified-billing-across-providers; (3) full observability via OpenTelemetry → Unity-Catalog-managed Delta tables joinable with HR/PR-velocity data — patterns/telemetry-to-lakehouse. Launch-day clients: Cursor, Codex CLI, Gemini CLI; Claude Code via MLflow 3 tracing. Structural mirror of the Cloudflare internal AI engineering stack shape with different substrates. Extends the wiki's existing AI-gateway provider- abstraction pattern along two new axes: coding-tool clients as first-class, and MCP-server governance as a peer concern to LLM-call governance. Introduces systems/unity-ai-gateway, systems/databricks-foundation-model-api, systems/cursor, systems/claude-code, systems/codex-cli, systems/gemini-cli, concepts/coding-agent-sprawl, concepts/centralized-ai-governance, patterns/central-proxy-choke-point, patterns/telemetry-to-lakehouse, patterns/unified-billing-across-providers. Tier-3 Databricks — ingested for framing + integration architecture, not internals; product-announcement post, gateway routing/fallback/rate-limiter mechanics not disclosed.)
• 2026-04-20 — sources/2026-04-20-databricks-take-control-customer-managed-keys-for-lakebase-postgres (Customer-Managed Keys rollout for systems/lakebase — Databricks' serverless Neon-descended Postgres. Three-level concepts/envelope-encryption hierarchy CMK → KEK → DEK; CMK held in customer's cloud KMS (systems/aws-kms / systems/azure-key-vault / systems/google-cloud-kms); concepts/cryptographic-shredding on revocation across both persistent (Pageserver+Safekeeper) and ephemeral (Postgres compute VM) layers; patterns/per-boot-ephemeral-key pattern for VM-local state; seamless key rotation as a property of the envelope hierarchy; Account↔Workspace delegation for separation of duties; auditability in customer KMS tenancy. Enterprise tier. Introduces systems/lakebase, systems/pageserver-safekeeper, systems/aws-kms, systems/azure-key-vault, systems/google-cloud-kms, concepts/envelope-encryption, concepts/cmk-customer-managed-keys, concepts/cryptographic-shredding, patterns/per-boot-ephemeral-key.)
• 2026-04-20 — sources/2026-04-20-databricks-mercedes-benz-cross-cloud-data-mesh (Mercedes-Benz cross-cloud data mesh on Unity Catalog + Delta Sharing + Delta Deep Clone; AWS Iceberg-on-Glue producer ↔ Azure Delta-on-ADLS consumers; hybrid "live share vs. incremental replica" tier; 66 % egress cost reduction on first 10 data products, ~93 % projected annual at 50 use cases; weekly-load → every-second-day freshness; DDX self-service orchestrator, DABs + Azure DevOps deploys, Sync-Job-bytes → producer chargeback, VACUUM for GDPR delete propagation on replicas; introduces systems/delta-sharing, systems/delta-lake, systems/mercedes-benz-data-mesh, concepts/data-mesh, concepts/egress-cost, concepts/hub-and-spoke-governance, concepts/cross-cloud-architecture, patterns/cross-cloud-replica-cache, patterns/chargeback-cost-attribution.)
• 2026-01-13 — sources/2026-01-13-databricks-open-sourcing-dicer-auto-sharder (open-sourcing Dicer, Databricks' auto-sharder; dynamic slice-range sharding with hot-key isolation/replication, eventually-consistent Assignments, state transfer across reshards; positioned vs. prior art Slicer / Centrifuge / Shard Manager; three production case studies — Unity Catalog 90–95% hit rate, SQL Query Orchestration zero-downtime scaling, Softstore 85% hit rate across rolling restarts via state transfer; use cases include LLM KV cache / LoRA-adapter GPU placement, batch aggregation, soft leader selection, rendezvous coordination.)
• 2025-12-03 — sources/2025-12-03-databricks-ai-agent-debug-databases (internal AI agent platform Storex for DB debugging across thousands of instances / 3 clouds / hundreds of regions / 8 regulatory domains; central-first sharded foundation; DsPy-inspired tools-as-functions framework; snapshot-replay validation with judge LLM; specialized per-domain agents; hackathon → platform journey; claimed up to 90% investigation-time reduction, <5 min new-hire ramp-up.)
• 2025-10-01 — sources/2025-10-01-databricks-intelligent-kubernetes-load-balancing (proxyless client-side L7 LB + custom xDS EDS; P2C + zone-affinity with spillover; rejected Istio / headless services; 20% pod-count reduction; surfaced cold-start problem that long-lived L4 LB had hidden.)

Ingest posture

Tier-3 filter applies: by default skip product PR, acquisition news, pure ML methodology posts. Ingest when the article covers: distributed-systems internals, scaling trade-offs, Kubernetes / network infrastructure, production incidents, storage/streaming design, or data-platform internals (Photon, Delta Lake, Unity Catalog — when architecturally substantive). Several 2025 posts already reviewed and logged as off-topic in log.md (TAO LLM-tuning, Neon acquisition PR, Data Intelligence for Marketing launch).