System Design — Overview¶
Synthesis across the wiki corpus. Snapshot (2026-08-03): 608 sources / 41 companies / 1,764 systems / 3,101 concepts / 2,007 patterns. Last ingest: 2026-08-01 (GitHub case-folding at memory speed). Growth rate: ~12–18 sources/week during active ingest; heavy backfill April 2026 (~145 articles).
Corpus shape¶
| Dimension | Count | Notes |
|---|---|---|
| Ingested sources | 608 | ~395 published 2026, ~140 in 2025, ~57 in 2024, ~16 older |
| Company pages | 41 | Top-5 by source count: Cloudflare (74), Databricks (57), Redpanda (42), AWS (41), Netflix (39) |
| Distinct system pages | 1,764 | Database clusters dominate (MySQL/Vitess/Postgres/Aurora/Lakebase) |
| Distinct concept pages | 3,101 | |
| Distinct pattern pages | 2,007 | |
| Cross-linked pages | ~6,900+ | Pages containing at least one [[wiki-link]] |
Company coverage (by wiki/sources/ count + dominant themes)¶
| Company | Sources | Tier | Dominant themes |
|---|---|---|---|
| Cloudflare | 74 | 1 | Edge compute (Workers), AI agents, PQC (auth+encryption), consensus (Meerkat), DDoS, caching, BGP |
| Databricks | 57 | 3→sel. | Lakehouse/Lakebase, Unity Catalog + AI Gateway, agent governance, Spark RTM, database branching |
| Redpanda | 42 | 2 | Kafka-compatible streaming, Iceberg/Cloud Topics, CDC, tiered storage, compaction, DR |
| AWS | 41 | 1 | S3 (Files, vectors), Lambda (1M functions), sovereignty/DR, CloudFront, EKS/Karpenter, multi-region |
| Netflix | 39 | 1 | ML serving (GenRec LLM-native recs), service topology, streaming codecs (AV1), Cassandra, workflow (Maestro), eBPF |
| Meta | 34 | 1 | GPU infra at scale, PQC migration, kernel scheduling (sched_ext), privacy infra, Rust/WhatsApp, AI storage |
| Fly.io | 31 | 3 | Micro-VMs (Firecracker), LiteFS/Litestream (SQLite replication), Rust proxy, GPU edge, Macaroons |
| PlanetScale | 28 | 3 | Vitess, MySQL internals, online DDL, consensus algorithms (8-part series), sharding, throttling |
| Figma | 23 | 3 | WebGPU rendering, multiplayer CRDT, caching (FigCache), sandboxing, custom DSLs, Postgres at scale |
| Zalando | 22 | 2 | SRE (operation-based SLOs), Kubernetes (route-server), Flink stream processing, load balancing (Skipper) |
| 20 | 1 | PQC, scheduling (time-varying capacity), privacy analytics, speculative inference, edge NPU | |
| 16 | 2 | ML recommenders, quota management (Piqama), deduplication (MiQPS), schema evolution, MCP ecosystem | |
| Slack | 14 | 2 | Shipyard (EC2 platform), deploy safety, notifications rebuild, AI multi-cloud, security agents |
| Instacart | 14 | 2 | On-call AI (Blueberry), ads retrieval, semantic IDs, LLM-at-scale batch (Maple), marketing platform |
| MongoDB | 14 | 2 | TLA+ verification, predictive auto-scaling, hybrid search, AI tool registries |
| Dropbox | 13 | 2 | Magic Pocket storage, Dash (AI search), Riviera (content processing), feature store |
| Airbnb | 13 | 2 | Metrics (fault-tolerant, OTel migration), Skipper workflow, data mesh (Viaduct), identity graph |
| Yelp | 12 | 3 | NRTSearch (Lucene), server-driven UI, S3 access logs at scale, Cassandra 4.x upgrade |
| Vercel | 12 | – | CDN (Bloom filter routing, request collapsing), agent-friendly pages, WebStreams perf |
| Atlassian | 11 | 2 | Events Rail (10B webhooks/mo), StreamHub (Kinesis→Kafka migration), OpenSearch, AI agents (Rovo) |
| GitHub | 8 | 2 | High-availability search, eBPF deployment safety, performance (case-folding, diff rendering) |
Top recurring architectural themes¶
1. AI/agent infrastructure is the dominant category (~35% of 2026 sources)¶
Every major company is shipping dedicated agent infra. Three architectural layers are converging:
- Execution substrate: concepts/durable-execution for multi-turn agent loops — Cloudflare (Project Think, Dynamic Workflows), Airbnb (Skipper), Instacart (Blueberry), Fly.io. Sagas + compensation for rollback.
- Governance/observability: systems/unity-catalog + systems/unity-ai-gateway for agent tool-use auditing and spend control (per-team budgets with daily+monthly caps). Datadog Bits AI SRE eval, Grafana Assistant context-learning, Databricks OTel+Unity for any-agent-anywhere observability.
- Serving at scale: Netflix in-house LLM serving (vLLM + custom routing + GenRec LLM-native recommendation), Databricks 200K QPS inference platform, Cloudflare inference layer for XL models, prompt caching for open-source models.
- Authorization: Databricks' concepts/intent-based-authorization (Omnigent) — agents declare intent, system validates against policy. Distinct from static RBAC.
- Self-service infra for agents: Databricks' infrastructure vending machine — agents can provision scoped resources with TTL-based lifecycle management and approval workflows.
- Cost control: Unity AI Gateway Budgets — dual-limit (daily + monthly) per-team spend caps with self-serve acknowledgment workflows and tiered overrides.
Key concepts: concepts/context-engineering, concepts/durable-execution, concepts/agent-governance-via-identity, concepts/intent-based-authorization, concepts/sub-agent, concepts/on-call-automation, concepts/agent-payment-budget-cap, concepts/prefill-only-inference.
2. Storage is converging: OLTP + OLAP + streaming under one governance layer¶
- Lakebase (Databricks): serverless Postgres on separated storage + compute, with concepts/database-branching, synced tables to lakehouse, LTAP transcoding, and agent orchestration via Postgres queues. 13+ sources.
- Redpanda Cloud Topics: Kafka-compatible streaming decoupled from local disk → S3 + metastore. Compaction, L0 GC, Little's Law applied. Single-query scaling validated to 1TB (strong scaling via scale-out).
- Apache Iceberg v3: unifying table format across Spark, Flink, Trino, Presto. Adopted by Databricks, Redpanda, AWS.
- Trend: the "lakehouse" pattern (Delta/Iceberg over object store) is absorbing what was previously separate OLTP, OLAP, and streaming tiers. R&D data now belongs in the same layer as operational data.
Key systems: systems/lakebase, systems/redpanda-cloud-topics, systems/apache-iceberg, systems/unity-catalog, systems/apache-spark, systems/redpanda-sql.
3. Post-quantum cryptography migration is accelerating (10+ sources, 4 companies)¶
Cloudflare targets full PQC by 2029 — now shipping PQ authentication to origins (ML-DSA signatures in mTLS), not just encryption. Meta has a fleet-wide crypto inventory + migration framework. Google disclosed quantum vulnerabilities via coordinated disclosure. GitHub shipped PQC for SSH.
The pattern emerging: encrypt first (ML-KEM), authenticate later (ML-DSA/SLH-DSA). Hybrid deployment (concepts/hybrid-key-encapsulation) is universal during transition. IPsec, TLS 1.3, SSH all have PQC paths in production. Signature size vs. latency remains the core tradeoff (ML-DSA at 2.4 KB vs. SQIsign compact but slow verify).
Key concepts: concepts/post-quantum-cryptography, concepts/post-quantum-authentication, concepts/crypto-agility, concepts/downgrade-attack.
4. Event-driven architecture at massive scale (webhooks, streaming, CDC)¶
Atlassian's Events Rail delivers 10B webhooks/month with per-tenant fairness and per-recipient concurrency limiting. StreamHub migrated from Kinesis to Kafka to handle 145B daily events. This pattern appears everywhere:
- Webhook/event delivery: Atlassian (Events Rail), Cloudflare (email service), AWS (EventBridge cross-region fan-out).
- Stream processing migration: Zalando migrated homegrown ad-event joins to Flink (75% state reduction via keyed CoProcess). Atlassian moved Kinesis→Kafka. Redpanda provides Kafka-compatible alternative.
- CDC as infrastructure primitive: Redpanda Connect (Oracle CDC GA), Datadog multi-tenant CDC platform, Databricks CDC pipelines into lakehouse, Segment S3-for-DynamoDB CDC.
Key patterns: patterns/two-stage-match-then-deliver-pipeline, patterns/per-tenant-rate-limiting, patterns/queue-isolation-per-pipeline, concepts/change-data-capture, concepts/backpressure.
5. Observability evolution: from dashboards to eBPF + topology + agents¶
- Netflix: real-time service topology map (~10 sources across service-topology, flow attribution, eBPF).
- Airbnb: StatsD→OTel migration; rebuilt monitoring ownership model.
- Datadog: multi-tenant CDC platform + eBPF runtime hardening.
- GitHub: eBPF for deployment safety verification.
- Lyft: metric semantic layer for self-serve analytics.
- Emerging: AI-powered observability — Grafana Assistant, Databricks OTel+Unity Catalog for agent tracing, Instacart Blueberry for on-call reasoning, Expedia service telemetry analyzer.
Key systems: systems/ebpf, systems/netflix-service-topology, systems/opentelemetry.
6. Edge computing + Workers as the default deployment target¶
Cloudflare alone has 74 sources; the Workers platform now hosts: durable objects, workflows, D1 (SQLite), R2 (object store), AI inference, Vectorize, Queues, email, DNS, Workers Cache, smart tiered cache, MoQ relay network — a full-stack platform at the edge. Vercel (12 sources) pushes similar patterns from the frontend-framework side (Bloom filter routing, request collapsing).
Key patterns: concepts/scale-to-zero, concepts/durable-execution, systems/cloudflare-workers, systems/cloudflare-smart-tiered-cache, systems/cloudflare-moq-relay-network.
7. Consensus + coordination keeps advancing¶
- Cloudflare's systems/meerkat: leaderless global consensus for control-plane data.
- PlanetScale's 8-part consensus-at-scale series (Vitess): establishment, revocation, race handling, propagation, completion.
- Netflix: interval-aware caching for Druid (coordination-free freshness).
- Redpanda: Raft-based internal + Shadow Link for DR replication.
Key concepts: concepts/consensus-algorithm, concepts/leaderless-consensus, concepts/linearizability, concepts/wide-area-consensus.
8. Database internals remain deeply documented¶
- PlanetScale (28 sources): MySQL adaptive hash index, memory profiling, throttler anatomy (3 parts), online schema migrations, connection pooling (PgBouncer), consensus at scale.
- Netflix: dynamic partition splitting in Cassandra for time-series, distributed counter abstraction, KV data abstraction layer.
- Figma: scaling Postgres horizontally (custom permissions DSL, FigCache, database team scale story).
- Zalando: Debezium logical replication at scale, JDBC driver patching, Flink stateful stream processing.
- Pinterest: automated schema evolution for next-gen DB, HBase deprecation.
- Atlassian: online index migration and shard-scaling in OpenSearch with AOSC coordinator.
9. Reliability engineering patterns are converging across companies¶
| Pattern | Companies using it |
|---|---|
| concepts/cell-based-architecture | AWS (Lakebase, CloudFormation), Cloudflare, Slack, Atlassian |
| concepts/blast-radius reduction | Netflix, AWS, Slack (Shipyard), Cloudflare (Code Orange), Atlassian (StreamHub cluster sharding) |
| patterns/canary-deployment / progressive | Slack, Netflix, Airbnb, GitHub |
| concepts/chaos-engineering | Netflix (Simian Army), Lakebase (failpoint injection), Fly.io |
| Incident response codification | Cloudflare (Code Orange), Slack (deploy safety), Instacart (Blueberry) |
| Immutable infrastructure | Slack (Shipyard bake-heavy), Netflix, AWS |
| Per-tenant fairness | Atlassian (Events Rail), Redpanda (adaptive write scheduling), Pinterest (Piqama) |
10. Low-level performance engineering resurfaces¶
GitHub's case-folding at memory speed (branchless SIMD, SWAR techniques, auto-vectorization) and Netflix's JDK Vector API for recommendation systems signal renewed interest in CPU-level optimization. This complements the GPU-focused work at Meta (MTIA, sched_ext) and Cloudflare (ClickHouse query plan contention).
Key concepts: concepts/simd-vectorization, concepts/branchless-programming, concepts/memory-bandwidth-bound, concepts/swar-byte-parallelism.
Most-linked pages (by inbound [[link]] count across corpus)¶
| Page | Inbound links | Category |
|---|---|---|
| systems/mysql | 358 | system |
| systems/vitess | 283 | system |
| concepts/blast-radius | 252 | concept |
| systems/aws-s3 | 215 | system |
| systems/redpanda | 196 | system |
| systems/postgresql | 187 | system |
| systems/kafka | 187 | system |
| concepts/observability | 186 | concept |
| systems/model-context-protocol | 170 | system |
| systems/lakebase | 164 | system |
| patterns/upstream-the-fix | 159 | pattern |
| systems/unity-catalog | 157 | system |
| concepts/control-plane-data-plane-separation | 156 | concept |
| concepts/llm-as-judge | 146 | concept |
| systems/apache-iceberg | 144 | system |
| systems/kubernetes | 142 | system |
| systems/cloudflare-workers | 140 | system |
Most-cited systems (by source references)¶
| System | Source refs | Theme |
|---|---|---|
| systems/aws-s3 | 53 | Object storage substrate for everything |
| systems/redpanda | 47 | Kafka-compatible streaming |
| systems/model-context-protocol | 47 | Agent tool interface standard |
| systems/mysql | 43 | Relational (via PlanetScale/Vitess deep dives) |
| systems/apache-iceberg | 41 | Open table format |
| systems/kafka | 38 | Streaming backbone |
| systems/cloudflare-workers | 36 | Edge compute |
| systems/postgresql | 34 | OLTP (via Lakebase, PlanetScale, Figma) |
| systems/kubernetes | 30 | Container orchestration |
| systems/unity-catalog | 30 | Data/AI governance |
| systems/lakebase | 27 | Serverless Postgres (Databricks) |
| systems/fly-machines | 27 | Micro-VM runtime |
Notable contradictions & open questions¶
-
Durable execution: Temporal-style event-sourcing vs. Airbnb's state-field replay. Temporal enforces determinism; Skipper trades auditability for zero-coordinator overhead. Neither has clearly "won" — context determines choice.
-
Kafka compaction correctness. Redpanda (June 2026) demonstrated that Kafka's log compaction can produce replica divergence via transaction control batches. Apache Kafka team has not yet responded. See sources/2026-06-25-redpanda-kafkas-log-compaction-corrupts-data.
-
PQC signature size vs. latency. ML-DSA is 2.4 KB per signature; SLH-DSA is 7-49 KB. TLS handshake bloat is real. Cloudflare explores SQIsign (compact but slow verify) and FN-DSA (fast but side-channel-sensitive). No clear winner for authentication yet.
-
Multi-tenant vs. dedicated infra for agents. Cloudflare pushes per-tenant dynamic code dispatch; Stripe/Databricks prefer dedicated databases per agent session. The cost/isolation tradeoff is unresolved.
-
Lakehouse vs. purpose-built OLTP. Databricks' thesis is convergence (Lakebase + lakehouse under one governance). Traditional DB vendors (PlanetScale, MongoDB) maintain purpose-built is better for transactional workloads. Data is sparse — Lakebase is <1.5 years old.
-
Agent authorization model. Databricks' intent-based approach (declare what you want, policy engine validates) vs. traditional RBAC/ABAC (static role grants). Both are in production; industry hasn't converged.
-
Kinesis vs. Kafka for event streaming. Atlassian migrated 145B daily events from Kinesis to Kafka (StreamHub), citing shard management overhead and cost. AWS counters with managed MSK. No universal answer — depends on operational model preference.
-
LLM-native recommendation vs. traditional two-tower. Netflix GenRec uses full LLM (prefill-only scoring) for recommendations while Meta's SilverTorch pushes GPU-native ANN retrieval. Whether LLMs replace or complement embedding-based retrieval is open — cost/latency tradeoff differs at different traffic volumes.
Trends observed (2025→mid-2026)¶
- AI-infra adoption went from "companies experimenting" to "companies shipping dedicated serving + governance platforms" in 12 months. Netflix, Databricks, Cloudflare, Airbnb all have production inference/agent platforms documented. Agent spend control is now a first-class concern.
- LLM-native systems are emerging: Netflix GenRec uses prefill-only LLM inference for recommendation scoring (not just generation), treating the LLM as a ranking function with catalog-aware scoring heads.
- Streaming → lakehouse integration accelerated: Redpanda's Iceberg Topics, Databricks' Spark RTM (real-time mode), CDC-into-lakehouse pipelines are now standard architecture. Redpanda SQL demonstrates strong scaling to 1TB single-query.
- Post-quantum moved from "research" to "fleet-wide deployment plans with dates" (Cloudflare: 2029, Meta: framework published, GitHub: SSH shipped). Authentication (not just encryption) is now shipping.
- eBPF expanded from observability-only to scheduling (sched_ext at Meta), security (Datadog runtime hardening, GitHub deployment safety), and flow attribution (Netflix).
- SQLite-at-the-edge emerged as a real architecture (Fly.io's Litestream/LiteFS, Cloudflare D1) — not just prototyping.
- Kubernetes abstractions matured: Karpenter (auto-provisioning), Kueue (batch scheduling), vCluster (multi-tenant), Slack Shipyard (next-gen EC2 platform on K8s).
- MCP (Model Context Protocol) became the dominant agent-tool interface, referenced by 47 sources across Cloudflare, Fly.io, Dropbox, Slack, Pinterest — the fastest-adopted standard in this corpus.
- Flink adoption broadened: Zalando migrated from homegrown stream joins to Flink, joining Pinterest, Databricks (Spark RTM), and Redpanda in the structured streaming ecosystem.
- Webhook/event delivery at scale emerged as a distinct architectural pattern: Atlassian's Events Rail (10B/month, per-tenant fairness, SQS fanout) documents the playbook for multi-tenant event delivery.
- BGP security and internet resilience are increasingly documented: Cloudflare publishes quarterly disruption reviews, BGP origin attribute manipulation research, and DNSSEC failure analyses — treating internet infrastructure as a system design topic.
- Low-level CPU optimization renaissance: GitHub (branchless SIMD for case-folding), Netflix (JDK Vector API), Meta (custom kernel schedulers) — companies are returning to per-instruction-level performance work alongside higher-level architectural patterns.
Reader navigation guide¶
Data sparsity notes¶
- Uber: 0 sources ingested (HTML scraper pending); historically canonical for microservices, Schemaless, Cadence.
- LinkedIn: stub only; no sources yet.
- Stripe: 4 sources, heavily 2026-focused (agent commerce, Radar fraud); older infra (Sorbet, idempotency keys) not yet in wiki.
- Google: 20 sources skew toward research (PQC, scheduling, privacy); internal systems (Spanner, Borg, Colossus) covered only via High Scalability secondhand references.
- Shopify: 2 sources (React Native only); no backend architecture yet.
- Databricks: high raw article count (260+ in raw/) but many filtered at ingest due to Tier-3 selectivity rules; 57 made it to wiki.
- Atlassian: growing (11 sources); recent additions (Events Rail, StreamHub, OpenSearch AOSC) are high-signal infrastructure posts.
Generated 2026-08-03 by wiki-synth. Read 608 sources, 41 company pages, sampled top systems/concepts/patterns. Next refresh recommended after source count reaches ~650.