Skip to content

related: [systems/netflix-titus, systems/netflix-atlas, systems/metaflow, systems/netflix-runq-monitor, systems/netflix-maestro, systems/netflix-sel, systems/netflix-amber, systems/netflix-fpd-cea, systems/netflix-data-gateway, systems/netflix-kv-dal, systems/netflix-timeseries-abstraction, systems/netflix-distributed-counter, systems/evcache, systems/netflix-media-production-suite, systems/netflix-content-hub, systems/netflix-open-connect, systems/netflix-footage-ingest, systems/netflix-flowexporter, systems/netflix-flowcollector, systems/netflix-ipman, systems/netflix-metatron, systems/netflix-sonar, systems/netflix-zuul, systems/netflix-data-mesh, systems/netflix-uda, systems/netflix-upper, systems/netflix-pdm, systems/netflix-sphere, systems/netflix-enterprise-graphql-gateway, systems/netflix-domain-graph-service, systems/av1-codec, systems/netflix-simian-army, systems/netflix-chaos-monkey, systems/netflix-latency-monkey, systems/netflix-conformity-monkey, systems/netflix-doctor-monkey, systems/netflix-janitor-monkey, systems/netflix-security-monkey, systems/netflix-10-18-monkey, systems/netflix-chaos-gorilla, concepts/data-contract, concepts/hybrid-cloud-media-ingest, concepts/open-media-standards, concepts/perceptual-conform-matching, concepts/ip-attribution, concepts/heartbeat-based-ownership, concepts/workload-identity, concepts/discrete-event-vs-heartbeat-attribution, concepts/tcp-tracepoint, concepts/amazon-time-sync-attribution, concepts/cross-regional-attribution-trie, concepts/knowledge-graph, concepts/domain-model, concepts/metamodel, concepts/named-graph, concepts/rdf, concepts/shacl, concepts/upper-ontology, concepts/conservative-extension, concepts/semantic-interoperability, concepts/data-container, concepts/film-grain-synthesis, concepts/auto-regressive-grain-model, concepts/grain-intensity-scaling-function, concepts/denoise-encode-synthesize, concepts/chaos-engineering, concepts/random-instance-failure-injection, concepts/availability-zone-failure-drill, concepts/graceful-degradation, patterns/chargeback-cost-attribution, patterns/data-abstraction-layer, patterns/sliding-window-rollup-aggregation, patterns/centralized-cloud-media-library, patterns/standards-driven-automation, patterns/heartbeat-derived-ip-ownership-map, patterns/sidecar-ebpf-flow-exporter, patterns/ebpf-map-for-local-attribution, patterns/kafka-broadcast-for-shared-state, patterns/regional-forwarding-on-cidr-trie, patterns/accept-unattributed-flows, patterns/model-once-represent-everywhere, patterns/self-referencing-metamodel-bootstrap, patterns/schema-transpilation-from-domain-model, patterns/graph-walk-sql-generation, patterns/decoder-side-synthesis-for-compression, patterns/codec-feature-gradual-rollout, patterns/continuous-fault-injection-in-production, patterns/simian-army-shape]

Netflix

The Netflix TechBlog (netflixtechblog.com) is a Tier-1 source on the sysdesign-wiki. Netflix runs one of the longest- running high-signal engineering blogs in the industry, covering streaming / CDN, container platforms, ML infrastructure, observability, video codecs, chaos engineering, and storage.

The RSS poller (see raw/_feeds.yaml) backfills the feed; per the current companies/index.md summary there are ~26 raw Netflix articles queued for ingestion as of 2026-04-22.

Key systems

Live streaming VBR cutover + MediaLive (2026-04-02 Live VBR post)

  • systems/aws-elemental-medialive — AWS Elemental MediaLive is Netflix Live's encoder substrate; its QVBR (Quality-Defined Variable Bitrate) setting is Netflix's capped VBR implementation. First canonical wiki instance of MediaLive in a Netflix role.
  • systems/netflix-open-connect — fleet delivery substrate for Netflix Live. Post-cutover: ≈10% lower peak-minute traffic (direct OC capacity- planning win) + ≈15% lower average bytes (direct CDN + peering ISP efficiency win).

Apache Druid + interval-aware query cache (2026-04-06 post)

  • systems/apache-druid — Apache Druid, Netflix's real-time OLAP / time-series substrate. Scale: >10 trillion rows, up to 15M events/sec ingested. Powers live-show monitoring, dashboards, automated alerting, canary analysis, A/B test monitoring. First wiki ingest of Druid.
  • systems/netflix-druid-interval-cache — Netflix's experimental external caching layer in front of Druid for rolling-window dashboard queries. Decomposes time-series queries into granularity-aligned time buckets (1-min minimum) keyed in a map-of-maps with SHA-256 query-shape hash outer key + big-endian timestamp inner keys for lex-order range scans; assigns age-based exponential TTLs per bucket (5 s floor for <2-min-old buckets, doubling per minute, 1-hour ceiling); on partial hit assembles a cached contiguous prefix + one narrowed Druid fetch for the missing tail (patterns/partial-cache-hit-with-tail-fetch); negative-caches interior empty buckets but not trailing empty buckets (late-arrival exception); deployed as an intercepting proxy at the Druid Router. Storage: first wiki-documented KVDAL consumer use case beyond the KVDAL launch post, exercising per-inner-key TTLs + inner-key range scans. Scale example: one popular dashboard (26 charts × 64 queries / refresh × 30 viewers / 10 s) emitted ~192 queries/sec — now mostly cache hits. Production results (typical day): 82% of queries get ≥partial hit, 84% of result data served from cache, P90 ~5.5 ms; A/B experiment: ~33% drop in queries to Druid, ~66% P90 improvement, up to 14× result-bytes reduction. Declared experimental; long-term direction is upstreaming the capability into Druid natively as a Broker-level opt-in result cache.

Multimodal video search — Marken + fusion + Elasticsearch (2026-04-04 post)

  • systems/netflix-marken — Netflix's annotation service, the transactional persistence gate for ML-model output describing media content. Cassandra-backed; captures per-model annotations (character recognition, scene detection, embeddings, confidence scores) at high-throughput ingest with "data integrity and high-speed write throughput" as the only job. Stage 1 of the three-stage video-search pipeline.
  • systems/apache-cassandra — dual-role storage substrate in Netflix's video-search pipeline: (1) raw annotation store underneath Marken; (2) target store for enriched temporal-bucket records written back by the offline-fusion stage. Netflix: "written back to Cassandra as distinct entities … a highly optimized, second-by-second index of multi-modal intersections."
  • systems/kafka — offline-fusion trigger bus. Every Marken annotation write publishes a Kafka event that triggers an asynchronous fusion job; a second Kafka event triggers indexing of enriched buckets into Elasticsearch. Canonical wiki instance of patterns/offline-fusion-via-event-bus — Kafka's role is decoupling heavy intersection compute from ingest such that "complex data intersections never bottleneck real-time intake."
  • systems/elasticsearch — stage-3 search index. Each temporal bucket is a root document; per-modality annotations are nested children; _id is the composite (asset_id, time_bucket) making model re-runs idempotent via composite-key upsert. The nested shape preserves cross-annotation-within-same-bucket query semantics — "this hierarchical data model is precisely what empowers users to execute highly efficient, cross-annotation queries at scale."

Ranker — homepage recommendation service + JDK Vector API optimization (2026-03-03 post)

  • systems/netflix-ranker — "one of the largest and most complex services at Netflix" — powers the personalized homepage rows. Stub covers the video serendipity scoring hot path and its 7.5% → ~1% per-operator CPU optimization via the JDK Vector API. The post doesn't describe Ranker's end-to-end architecture (retrieval, candidate gen, ranking model) — stub explicit about scope.
  • systems/jdk-vector-api — pure-Java SIMD as an incubating JDK feature. DoubleVector.SPECIES_PREFERRED picks the widest host lane width at runtime (4 doubles on AVX2, 8 on AVX-512); fma() per-lane instruction; scalar fallback. No JNI, no native build. First canonical wiki instance from Netflix production.
  • systems/lucene — Apache Lucene's VectorUtilDefaultProvider is Netflix's inspiration for the scalar-fallback loop-unrolled dot product (author-credited to Patrick Strawderman).

MediaFM — multimodal foundation model for media understanding (2026-02-23 post)

  • systems/netflix-mediafm — Netflix's **first tri-modal (audio
  • video + timed-text) foundation model for media understanding. BERT-style Transformer encoder pre-trained with Masked Shot Modeling (MSM) — mask 20% of input shots, predict the original fused embedding at masked positions via cosine distance. Input: sequences of shot-level fused embeddings** (up to 512 shots per title), each fused by concatenating + unit-normalising three per-modality vectors — SeqCLIP for video, Meta FAIR wav2vec2 for audio, OpenAI text-embedding-3-large for timed text (closed captions / audio descriptions / subtitles; zero-padded when absent). Two special tokens prepended to every sequence: a learnable [CLS] and a [GLOBAL] token built from title-level metadata (synopses
  • tags). Optimisation: Muon for hidden parameters, AdamW for the rest; the Muon switch is flagged as "noticeable improvements" without numerical ablation. Frozen after pre-training, evaluated + deployed via task-specific linear probes on five Netflix downstream tasks — ad relevancy (AP), clip popularity ranking (10-fold Kendall's τ), clip tone (100-category micro AP), clip genre (11-category macro AP), clip retrieval (binary "clip-worthy", 1:3 pos:neg, AP). MediaFM beats all baselines on all five. Ablation finding: contextualisation — not additional modalities — delivers most of the gain, especially on narrative- understanding tasks; uncontextualised tri-modal concat actually hurts clip popularity ranking vs single-modality baseline. Deployment rule: "embedding in context" — embed a short clip by running MediaFM on its full containing title and slicing out the clip-span vectors; running on the clip alone is materially worse. Production consumers: ad-relevancy retrieval stage, clip tagging, optimised promotional assets (art + trailers), internal content-analysis tools, and cold start of newly-launching titles in recommendations (content-derived embedding ready at launch, no user-interaction data needed). Forward direction: investigate swapping pre-trained multimodal LLMs like Qwen3-Omni in place of the current fuse-yourself approach.
  • systems/netflix-seqclip — Netflix-internal CLIP-style video encoder fine-tuned on video retrieval datasets, used as MediaFM's frozen video-modality sub-encoder (embeds frames sampled at uniform intervals from each shot). Descendant of OpenAI's CLIP.

Simian Army — chaos engineering (2011 foundational post)

  • systems/netflix-simian-army — umbrella for Netflix's fleet of narrowly-focused automated agents that continuously exercise fault-tolerance in AWS production. Canonical origin of chaos engineering as a production discipline (the term "chaos engineering" was coined ~2016; the practice was declared here in 2011). Eight named simians, each owning one failure-mode or abnormal-condition domain.
  • systems/netflix-chaos-monkey — randomly terminates production instances. Founding member. Runs in business hours, under engineer supervision. Canonical concepts/random-instance-failure-injection instance.
  • systems/netflix-latency-monkey — injects artificial RPC-boundary delays; modest delays test degradation, large delays simulate outage without instance teardown. More surgical than Chaos Monkey for testing a new service against simulated dependency failure.
  • systems/netflix-chaos-gorilla — simulates full AWS availability-zone outage; verifies automatic re-balance without user impact or manual intervention. Canonical concepts/availability-zone-failure-drill instance.
  • systems/netflix-conformity-monkey — drift detector for operational best-practices (e.g. instance not in an ASG); enforces by termination.
  • systems/netflix-security-monkey — "extension of Conformity Monkey" for security drift: mis-configured AWS security groups, expiring SSL / DRM certificates. Ancestor of the later open-source Netflix/security_monkey platform.
  • systems/netflix-doctor-monkey — unhealthy-instance detector; two-phase eviction (remove-from-service → eventually terminate) allows owners to root-cause before cleanup.
  • systems/netflix-janitor-monkey — unused-resource cleanup; cost-and-hygiene axis. Ancestor to FPD + CEA cloud-efficiency platform.
  • systems/netflix-10-18-monkey — l10n / i18n drift detector for configuration + runtime problems across geographies / languages / character sets. Least-specified monkey in the 2011 post.

Linux performance triage toolbox (2025-07-29 60-second checklist post)

  • systems/vmstat — BSD-vintage (1980s) virtual-memory statistics tool. Load-bearing columns on one line: r (CPU saturation), us/sy/id/wa/st (CPU time breakdown), si/so (swap).
  • systems/iostat — per-block-device I/O statistics via iostat -xz 1; %util / avgqu-sz / await for saturation + latency.
  • systems/mpstat — per-CPU breakdown via mpstat -P ALL 1; exposes single-hot-CPU patterns invisible to vmstat's system-wide averages.
  • systems/pidstat — per-process CPU/mem/I/O via pidstat 1; rolling output (not top's clear-screen) ideal for incident capture.
  • systems/sar-sysstat — System Activity Reporter via sar -n DEV 1 (NIC bytes/pps) + sar -n TCP,ETCP 1 (active / passive / retrans). Also archive mode via sadc for historical counters going back days / weeks.
  • systems/linux-top — interactive per-process snapshot; 10th command in the checklist as the sanity-check catch-all.
  • systems/sysstat-package — umbrella package that ships sar / iostat / mpstat / pidstat. Operationally: include in base AMI.

AV1 video codec + Film Grain Synthesis (2025-07-03 AV1-FGS post)

  • systems/av1-codec — AOMedia's royalty-free video codec. On this wiki, first documented role is the decoder target for Netflix's at-scale rollout of Film Grain Synthesis in 2025-07; Netflix first shipped AV1 on TVs in 2021 but enabled FGS only on "a limited number of titles" at that launch. The 4-year gap 2021 → 2025 was rollout engineering — encoder-side denoiser, grain-parameter estimation, quality evaluation, device compatibility on the long tail of AV1 decoders — not AV1 spec work. The AV1 standard defines the grain-parameter format + the decoder-side synthesis procedure but does not specify the encoder denoiser, leaving that to per-vendor investment.

eBPF flow-log attribution (2025-04-08 IP-attribution post)

  • systems/netflix-flowexporter — per-host eBPF sidecar attached to TCP tracepoints; emits a flow log on each socket close with local workload identity pre-resolved. ~5M records/sec fleet-wide, 1-minute batch reporting.
  • systems/netflix-flowcollector — regional backend attribution service on 30 c7i.2xlarge processing 5M flows/sec with no persistent storage; maintains in-memory per-IP time-range map; Kafka broadcast to peer nodes; 1-minute disk buffer for remote attribution; CIDR-trie forwarding for cross-region flows.
  • systems/netflix-ipman — container IP assignment service; IPManAgent daemon writes IP → workload-ID into an eBPF map that FlowExporter's BPF programs read in-kernel.
  • systems/netflix-metatron — EC2-instance-level workload identity provisioner (certs at boot, read from local disk).
  • systems/netflix-sonar — legacy discrete-event IP-tracking service; retained only for ELB / non-workload IP attribution where heartbeat-based attribution is impossible.
  • systems/netflix-zuul — cloud gateway; load-bearing ground- truth validation target (routing config → expected dependencies); baseline ~40% misattribution under the old system → 0 in the new system over a 2-week validation window.
  • systems/netflix-data-mesh — downstream stream/batch processing platform consuming attributed flows.

Data Gateway platform + three mature abstractions (2024-09-19 KV DAL + 2024-11-13 Counter posts)

  • systems/netflix-data-gateway — the platform layer hosting Netflix's Data Abstraction Layer services. Containers per abstraction, namespace-driven routing, composition between layers on the same host.
  • systems/netflix-kv-dal — mature gRPC service exposing a two-level-map data model over Cassandra + EVCache + DynamoDB + RocksDB.
  • systems/netflix-timeseries-abstraction — event store for temporal event data, Cassandra-backed with bucketed partitioning; the event store underneath the Counter service.
  • systems/netflix-distributed-counter — counting service built on top of TimeSeries + EVCache. ~75K req/s globally at single-digit-ms latency; Best-Effort (EVCache-only) vs Eventually-Consistent (event-log + background sliding-window rollup) taxonomy; experimental Accurate mode with real-time delta. Canonical wiki instance of one DAL consuming another.
  • systems/evcache — Netflix's distributed in-memory cache. Two roles in the Counter story: Best-Effort backing + Rollup Cache. Also the cache tier layered under KV DAL namespaces.

Media Production Suite (2025-04-01 MPS post)

  • systems/netflix-media-production-suite — cloud-based filmmaker toolchain inside Content Hub, covering the production lifecycle from on-set capture through picture finishing. Seven tools: Footage Ingest (gateway), Media Library, Dailies, Remote Workstations, VFX Pulls, Conform Pulls, Media Downloader. >350 titles across UCAN / EMEA / SEA / LATAM / APAC. Designed for ~200 TB-per-title OCF average / up to ~700 TB outliers. LTO tape creation is default-off under MPS.
  • systems/netflix-content-hub — parent production portal hosting MPS + Workspaces (Google-Drive-style shared folders used by VFX Pulls for vendor handoff) + the Footage Ingest remote- monitoring dashboard.
  • systems/netflix-footage-ingest — drive-plug-in gateway application. Six-stage pipeline: validate drive manifest → upload OCF + OSF → checksum → inspect + metadata extract → build playable proxies → tier-2 cloud archive. Every other MPS tool reads the library that Footage Ingest populates.
  • systems/netflix-open-connect — Netflix's CDN, here in its first documented non-streaming role: carrying ingest-centre ↔ AWS media traffic for MPS. First appearance of the canonical Netflix CDN on the wiki.

UDA — Unified Data Architecture (2025-06-14 UDA post)

Existing systems

  • systems/netflix-titus — Netflix's internal container platform (Kubernetes-based, with a "thick layer of enhancements over off-the-shelf Kubernetes" for observability, security, scalability, cost).
  • systems/netflix-atlas — primary telemetry / metrics platform (dimensional time-series DB, open-source).
  • systems/metaflow — ML framework (open-source; foundational layer + per-team domain libraries).
  • systems/netflix-maestro — Netflix-internal workflow orchestrator (replaces Step Functions / Argo / Airflow in the open-source Metaflow path). Substantially expanded on 2026-04-22 with the 2024-07-22 Maestro open-sourcing post — horizontally scalable single-cluster engine running ~500K jobs/day average / ~2M peak / 87.5% YoY growth; acyclic + cyclic workflows with foreach + subworkflow + conditional-branch composite primitives; five named run strategies; SEL-sandboxed parameterized workflows; seven-layer step parameter merging; signal-based step dependencies with exactly-once trigger guarantee + signal lineage; per-step breakpoints for in-flight debugging and state mutation; platform-vs-user retries with exponential backoff; eventually-consistent rollup across nested subworkflows + foreach.
  • systems/netflix-sel — Simple Expression Language; homemade JLS subset with loop / array / memory runtime limits + Java Security Manager sandbox; enables safe code injection in Maestro parameterized workflows.
  • systems/netflix-amber — media feature store; uses Metaflow Hosting for on-demand feature compute.
  • systems/netflix-runq-monitor — eBPF-based per-container run-queue-latency monitor running on Titus hosts.
  • systems/netflix-metaflow-fast-data · systems/netflix-metaflow-hosting · systems/netflix-metaflow-cache — Netflix-internal integrations layered on Metaflow.
  • systems/netflix-fpd-cea — Netflix's two-layer internal cloud-efficiency data platform: FPD (Foundational Platform Data) normalises inventory/ownership/usage per platform via data contracts; CEA (Cloud Efficiency Analytics) layers business-logic on FPD to produce attributed-cost time-series with single-owner resolution + multi-tenant distribution + multi-aggregation output. Documented publicly 2025-01-02 as the substrate powering FinOps decisions at Netflix.

Key patterns / concepts

Multimodal video search pipeline (2026-04-04 video-search post)

  • patterns/three-stage-ingest-fusion-index — canonical wiki instance: transactional persistence (Marken / Cassandra) → offline fusion (Kafka-triggered, bucket-discretize + cross-model intersection) → indexing-for-search (Elasticsearch nested documents, composite-key upsert). Netflix's framing: "Cleanly decoupling these intensive processing tasks from the ingestion pipeline guarantees that complex data intersections never bottleneck real-time intake."
  • patterns/offline-fusion-via-event-bus — Kafka-glued decoupling of heavy fusion compute from ingest; sibling of Netflix's Distributed Counter rollup-trigger pattern on the counter axis.
  • patterns/temporal-bucketed-intersection — three-step bucket-mapping / annotation-intersection / optimised-persistence algorithm. Worked example: "Joey" 2-8s × "kitchen" 4-9s → four shared one-second buckets.
  • patterns/nested-elasticsearch-for-multimodal-query — root asset + bucket identity, nested source_annotations children per modality. Preserves cross-annotation-within-parent semantics.
  • concepts/temporal-bucket-discretization — fixed-size-bucket discretization of continuous time annotations as the enabling primitive for multimodal temporal joins. One-second buckets in Netflix's worked example.
  • concepts/multimodal-annotation-intersection — cross-model co-occurrence in a shared bucket is the Ingest-time fusion semantic. Character recognition + scene detection + (implicitly) more modalities fused per bucket.
  • concepts/composite-key-upsert — (asset_id, time_bucket) as Elasticsearch _id for idempotent model re-runs. Third canonical Netflix instance after KV DAL's (generation_time, nonce) and Distributed Counter's (event_time, event_id, event_item_key).
  • concepts/nested-document-indexing — Elasticsearch nested field type for per-modality child documents; enables correct cross-annotation-within-parent queries that flat documents can't express.

Performance engineering — serendipity scoring optimization (2026-03-03 JDK Vector API post)

  • patterns/batched-matmul-for-pairwise-similarity — headline algorithmic reshape: turn O(M×N) per-pair cosine similarities into a single matmul C = A × Bᵀ. Canonical wiki instance via Netflix Ranker's video serendipity scoring.
  • patterns/flat-buffer-threadlocal-reuse — enabling substrate for SIMD: replace double[M][D] with flat row-major double[], wrap in ThreadLocal<BufferHolder> grow-but-never-shrink buffers. The first batched implementation regressed ~5% without this step.
  • patterns/runtime-capability-dispatch-pure-java-simd — deployment safety for incubating APIs: detect jdk.incubator.vector at class load, fall back to a high-quality scalar path (Lucene-inspired loop-unrolled dot product) when absent. Keeps the service safe if the --add-modules flag isn't set.
  • concepts/cosine-similarity — the per-pair kernel Netflix's matmul implements at batch granularity.
  • concepts/jni-transition-overhead — the reason BLAS lost the kernel competition. Per-call JNI transition + layout translation + temp-buffer allocation alongside upstream TensorFlow allocations ate the native-kernel speedup.
  • concepts/row-vs-column-major-layout — Java is row-major, classical BLAS / LAPACK is column-major. Translation forces conversions and temporary buffers; pure-Java SIMD sidesteps this.
  • Extends concepts/matrix-multiplication-accumulate with Netflix's CPU-side FMA-based C ← A × B + C instance on AVX-512 hardware, complementing the existing Tensor-Core-on-GPU framings.
  • Extends concepts/cache-locality with the flat-buffer + row-major-access enabler variant.
  • Extends concepts/flamegraph-profiling as the diagnostic that surfaced the 7.5% hot-path target.
  • Extends patterns/measurement-driven-micro-optimization with Netflix's five-step canary-validated sequence (nested loops → batched matmul regression → flat buffers → BLAS regression → JDK Vector API).

Chaos engineering — Simian Army (2011 foundational post)

  • concepts/chaos-engineering — canonical wiki definition of the discipline. Netflix's 2011 Simian Army post is the origin reference; the term "chaos engineering" was coined ~2016 for a practice Netflix had been running for five years. Claim: "just designing a fault tolerant architecture is not enough. We have to constantly test our ability to actually survive these 'once in a blue moon' failures."
  • concepts/random-instance-failure-injection — the Chaos Monkey primitive: pick a production instance at random and kill it, verify survival. Canonical wiki instance via systems/netflix-chaos-monkey.
  • concepts/availability-zone-failure-drill — the Chaos Gorilla primitive: simulate full AZ outage, verify automatic re-balance. Canonical wiki instance via systems/netflix-chaos-gorilla. Three success criteria — automatic re-balance, no user-visible impact, no manual intervention — are the AZ-failure tolerance contract.
  • concepts/graceful-degradation — prerequisite for chaos engineering. Netflix's 2011 framing pairs graceful degradation with node-/rack-/AZ-/region-redundant deployments as the designed side of the architecture; the Simian Army is the exercised side that keeps those designs honest. Canonical wiki definition.
  • patterns/continuous-fault-injection-in-production — the scheduling discipline: business hours + engineer supervision
  • production environment + continuous cadence. "By running Chaos Monkey in the middle of a business day … we can still learn the lessons about the weaknesses of our system." Cloud makes this pattern economically viable where physical datacenters can't.
  • patterns/simian-army-shape — the architectural-shape pattern: fleet of narrowly-focused agents, each owning one failure mode or one abnormal-condition domain, composed at the fleet level. Unifies fault injectors (Chaos / Latency / Gorilla) with drift detectors (Conformity / Security / Doctor / Janitor / 10-18). Canonical Netflix instance.

Linux performance triage (2025-07-29 60-second checklist post)

  • patterns/sixty-second-performance-checklist — canonical wiki pattern: 10 stock Linux commands run in a defined order as the first minute of any performance investigation. Errors + saturation before utilisation; hand-off to eBPF / flame graphs / Atlas afterwards.
  • patterns/utilization-saturation-errors-triage — the reusable enumeration discipline: for every resource, check utilisation + saturation + errors; exonerate as you go; don't advance to root cause until the sweep is complete.
  • concepts/use-method — Brendan Gregg's Utilisation/Saturation/Errors methodology; the 60-second checklist is its encoding as 10 shell commands.
  • concepts/load-average — demand signal (includes both runnable and uninterruptible-I/O-blocked tasks on Linux); "worth a quick look only" — use 1/5/15-min trend, then pivot to vmstat.
  • concepts/cpu-utilization-vs-saturation — two separate measurements on the same CPU; us+sy vs r. The most common triage mistake is conflating them.
  • concepts/cpu-time-breakdown — us / sy / id / wa / st as the diagnostic decomposition; sy > 20% worth investigating, %steal > 0 is the in-guest signature of hypervisor co-tenancy (concepts/noisy-neighbor).
  • concepts/io-wait — %iowait is CPU idle with a reason; points to disk, pivot to iostat -xz 1.
  • concepts/linux-page-cache — free -m's -/+ buffers/cache row is the load-bearing memory accounting; ZFS-on-Linux ARC is a further caveat free doesn't reflect.

eBPF flow-log attribution (2025-04-08 IP-attribution post)

Data Gateway + three mature abstractions (2024-09-19 KV DAL + 2024-11-13 Counter posts)

  • patterns/data-abstraction-layer — Netflix's load-bearing architectural shape: a gRPC DAL between microservices and storage engines that exposes a uniform data-problem vocabulary + routes per-namespace to the right backing stores. KV, TimeSeries, and Counter are three mature instances.
  • patterns/namespace-backed-storage-routing — namespace as the unit of logical + physical configuration; control-plane-driven.
  • patterns/sliding-window-rollup-aggregation — canonical wiki instance is Netflix Counter's Eventually-Consistent mode: TimeSeries event log + in-memory rollup queues + Cassandra Rollup Store + EVCache Rollup Cache, aggregating within an immutable window and checkpointing the result.
  • patterns/bucketed-event-time-partitioning — TimeSeries schema uses (time_bucket, event_bucket) columns to prevent Cassandra wide partitions under high event throughput; per-namespace tuning.
  • patterns/fire-and-forget-rollup-trigger — post-durability write path fires a light-weight rollup event to the rollup tier; reads also emit; last-write-timestamp is the independent self-healing signal.
  • concepts/event-log-based-counter — counter as an event log aggregated in the background, preserving audit + recounting + reset semantics over a naïve in-place counter.
  • concepts/best-effort-vs-eventually-consistent-counter — the two-mode taxonomy Netflix surfaces, plus an experimental Accurate mode with a real-time delta on top of the Eventually Consistent checkpoint.
  • concepts/immutable-aggregation-window — the concurrency-safety trick underneath the rollup pipeline; acceptLimit on the event store makes the aggregation window frozen by construction.
  • concepts/lightweight-rollup-event — signaling-only event (namespace + counter, no delta) that tells the rollup server a counter needs attention; routed by XXHash + coalesced per window.
  • concepts/idempotency-token — canonical Netflix instance, covering both KV DAL (generation_time, nonce) writes and Counter (event_time, event_id, event_item_key) events.
  • concepts/last-write-wins — Cassandra USING TIMESTAMP used operationally at the Counter Rollup Store on last-write- timestamp; the skew-bounded wall-clock LWW variant.

Observability + performance isolation (2024-09-11 noisy-neighbor eBPF post)

ML platform (2024-07-22 Diverse ML Systems post)

Workflow orchestration (2024-07-22 Maestro post)

Media production (2025-04-01 MPS post)

  • patterns/centralized-cloud-media-library — upload once to a cloud-addressable asset namespace; every downstream consumer (editorial, VFX, DI, archive, monitoring) reads from that single library. Replaces LTO-tape + hand-carried-drive distribution. Canonical wiki instance via Netflix MPS.
  • patterns/standards-driven-automation — choose public cross-vendor interchange standards (ACES / AMF / ASC MHL / ASC FDL / OTIO) over per-facility bespoke hot-folder scripts. Collapses automation effort from O(producers × consumers) to O(producers + consumers) + democratises access to complex workflows for emerging-market productions.
  • concepts/hybrid-cloud-media-ingest — infrastructure shape: edge ingest centres close to production sites + CDN-class backhaul (Open Connect) + AWS durable substrate. Necessary precondition for populating the centralised library fast enough at 200–700 TB per title.
  • concepts/open-media-standards — ACES + AMF (colour pipeline); ASC MHL (checksum/manifest); ASC FDL (framing interoperability); OTIO (timeline interchange). Each standard makes one workflow stage automatable at scale. Adjacent to data contracts — same coordination primitive, cross-company-ecosystem form vs. internal-team form.
  • concepts/perceptual-conform-matching — fallback hierarchy for resolving EDL → OCF references: exact metadata match → fuzzy metadata match → (future) perceptual CV match. Generalises to any cross-system reference-resolution pipeline that can fall through to content similarity.

UDA data integration + semantic interoperability (2025-06-14 UDA post)

  • patterns/model-once-represent-everywhere — headline UDA pattern. Promote the conceptual model from docs/tribal knowledge to a first-class control-plane artifact and project it outward into every schema / API / pipeline that needs to know about the concept. One authored source → many generated representations. Canonical wiki data-layer instance; siblings exist at the API-surface layer (patterns/schema-driven-interface-generation, Cloudflare) and the deploy-config layer (patterns/single-source-service-definition, Figma).
  • patterns/self-referencing-metamodel-bootstrap — the metamodel design pattern Upper embodies: self-referencing / self-describing / self-validating. Upper is its own first customer — its Java API + GraphQL schema are projected by UDA's transpiler family + federated into the Enterprise GraphQL Gateway on every change. The metamodel exercises the transpiler in production continuously.
  • patterns/schema-transpilation-from-domain-model — the transpiler-family pattern: one authored domain model → one transpiler per target language (GraphQL / Avro / SQL / RDF / Java) → generated schema + auto-provisioned data product + auto- provisioned pipeline + generated UI. Contrast with patterns/gradual-transpiler-migration (migration pattern, one-shot) and patterns/schema-driven-interface-generation (sibling at API-surface layer).
  • patterns/graph-walk-sql-generation — Sphere's concept-to-SQL mechanism: walk the knowledge graph from business concepts to data containers, emit SQL that runs against the warehouse natively. Knowledge graph is the planner, warehouse is the executor — a deliberate design response to SPARQL's historical scale limitations, though the post doesn't frame it that way.
  • concepts/knowledge-graph — second wiki framing added alongside the Dropbox-Dash agent-retrieval-substrate framing: Netflix UDA is the enterprise-data-integration substrate framing — the graph unifies schema registry + data catalog + transpiler source + pipeline source.
  • concepts/domain-model — canonical wiki definition via UDA's Upper-authored controlled vocabulary of keyed entities / attributes / relationships / taxonomies, treated as data (not code, not docs).
  • concepts/metamodel — the "model of models" framing. Upper is the wiki's canonical metamodel instance.
  • concepts/named-graph — RDF's modular-partition primitive; UDA's info model is named-graph-first — every named graph conforms to a governing named graph, all the way up to Upper.
  • concepts/rdf — canonical production RDF deployment on the wiki. UDA chose RDF + SHACL as the foundation; Upper enumerates the gaps UDA had to fill on top (no info-model guidance for named graphs; owl:imports only covers ontologies not data; enterprise local-keys + multi-graph patterns absent).
  • concepts/shacl — the shape-validation standard beneath UDA with an explicit enterprise-fit limitation ("SHACL is not a modeling language for enterprise data" — global-URI + single- data-graph assumptions don't match enterprise local-schema + typed-key patterns).
  • concepts/upper-ontology — production-enterprise upper ontology instance, distinct from the classical theoretical/ standardisation artefacts (BFO / DOLCE / SUMO).
  • concepts/conservative-extension — the formal composition- safety property UDA relies on: new domain models strictly add vocabulary + axioms without retracting prior facts, guaranteed by Upper's design. The algebraic analog of backward-compatible schema evolution.
  • concepts/semantic-interoperability — the load-bearing requirement that pushed UDA's design towards a knowledge graph over RDF + SHACL. Without it, schema-registry-only deployments still end up with "same schema, different meanings" drift.
  • concepts/data-container — UDA's unifying abstraction for the many heterogeneous places instance data lives (federated GraphQL entities / Avro / Iceberg rows / Java API objects). Containers are both projection targets + graph-representation sources + pipeline endpoints.
  • Extended concepts/schema-registry — UDA is the wiki's canonical instance of a schema registry + data catalog unified into one substrate (the knowledge graph), distinct from the prior Amazon Key / EventBridge framing of schema registry as a stand-alone service.

Key patterns / concepts

Interval-aware caching — rolling-window dashboards at hyperscale (2026-04-06 Druid cache post)

  • patterns/interval-aware-query-cache — headline pattern. Decompose time-series queries into granularity-aligned buckets + age-based exponential TTLs
  • contiguous-prefix lookup with one narrowed backend fetch. Netflix's Druid cache is the canonical wiki instance; the post explicitly flags the pattern as non-Druid-specific — "splitting time-series results into independently cached, granularity-aligned buckets with age-based exponential TTLs isn't Druid-specific and could apply to any time-series database with frequent overlapping-window queries."
  • patterns/age-based-exponential-ttl — sub-pattern. TTL scales monotonically with data age: 5 s floor for <2-min-old buckets, doubling per additional minute, capped at 1 hour. Fresh buckets cycle fast (late-arriving corrections); old buckets linger (confidence grows with time).
  • patterns/partial-cache-hit-with-tail-fetch — sub-pattern. Contiguous-prefix scan from interval start; on first gap, stop and fetch the entire missing tail in one narrowed backend query. Fewer backend queries > narrower queries — query setup cost dominates per-bucket scan cost.
  • patterns/intercepting-proxy-for-transparent-cache — deployment shape. External cache intercepts at the Druid Router, falls through for non-cacheable requests, back-through-the-Router for cache misses. Zero client changes. Netflix frames the external proxy as a temporary posture — long-term direction is upstreaming into Druid proper.
  • concepts/rolling-window-query — the workload shape that makes the cache useful: [now - Δ, now] queries that refresh with a shifting right boundary.
  • concepts/granularity-aligned-bucket — the cache-layer decomposition unit; fixed-size query-granularity-aligned time buckets are the atomic reusable cache entry.
  • concepts/exponential-ttl — the concept page for the TTL strategy.
  • concepts/negative-caching — caching empty sentinel values for naturally sparse metrics, with the trailing-bucket exception (empty trailing buckets aren't cached — they might just be late-arriving data).
  • concepts/late-arriving-data — the forcing function behind age-based TTLs + trailing-bucket exception; Netflix's pipeline P90 <5 s bounds the cache's 5 s floor.
  • concepts/query-structure-aware-caching — the cache parses queries and decomposes responses along a structural axis (time) rather than treating them as opaque blobs.
  • concepts/time-series-bucketing — the general framing; Druid segments, Netflix Distributed Counter rollup buckets, and the interval-aware cache all bucket time differently at different layers of the stack.
  • concepts/staleness-vs-load-tradeoff — the declared architectural trade-off. Canonical wiki framing of "bounded staleness in exchange for bounded backend load" with the explicit pipeline-latency-vs-TTL comparison: Netflix's 5 s cache TTL is ~= pipeline P90 ingestion lag, so the cache adds negligible staleness on top of what's already there.

Video codec tools + decoder-side synthesis (2025-07-03 AV1-FGS post)

  • concepts/film-grain-synthesis — AV1 codec tool that strips film grain from the source before compression, transmits a compact parameter set (AR coefficients + piecewise-linear scaling function), and re-synthesizes the grain on the decoder. Canonical instance on the wiki.
  • concepts/auto-regressive-grain-model — AR model for the grain pattern component; a handful of coefficients drive generation of a 64×64 noise template, from which random 32×32 patches are tiled onto decoded frames. "a linear combination of previously synthesized noise sample values, with AR coefficients a₀, a₁, a₂, a₃ and a white Gaussian noise (wgn) component."
  • concepts/grain-intensity-scaling-function — piecewise-linear function mapping pixel value → grain intensity; models the empirical fact that film grain is more visible in mid-tones than in blacks/highlights. "the film grain strength is adapted to the areas of the picture".
  • concepts/denoise-encode-synthesize — three-stage encoding- pipeline shape induced by FGS: denoise the source (vendor choice, not standardised), encode the clean signal, transmit AR coefficients + scaling function as side channel, re- synthesize grain on the decoder. Extends concepts/video-transcoding with the synthesis-based variant distinct from the Meta-FFmpeg-scale multi-encoder-lane shape.
  • patterns/decoder-side-synthesis-for-compression — the architectural pattern generalised: transmit parameters of a generator, not the signal itself. Canonical production instance on the wiki is AV1 FGS. The main bitstream carries a codec-friendly residual (denoised video); a small side channel carries generator parameters; the decoder reconstructs the component locally. Wins when the component is high- entropy + statistically describable + perceptually tolerant of substitution + cheap to synthesize — all four true for film grain. Reference metrics (VMAF / PSNR / SSIM) break down because the output is sample-wise different from the source even when perceptually equivalent — extends concepts/visual-quality-metric.
  • patterns/codec-feature-gradual-rollout — Netflix's 2021 → 2025 FGS rollout is the canonical wiki instance. A codec feature can be standardised years before it is deployable at scale; the delta is per-vendor encoder tooling, device- compatibility testing across the long tail of deployed decoders, quality-evaluation methodology, and encoding-ladder integration. Staged rollout bounds blast radius + lets the deployed-decoder denominator grow while encoder investment pays off.

Cloud efficiency / FinOps (2025-01-02 Cloud Efficiency at Netflix post)

  • systems/netflix-fpd-cea — two-layer internal data platform: FPD (Foundational Platform Data) normalises inventory/ownership/usage per platform via data contracts; CEA (Cloud Efficiency Analytics) layers business-logic to produce attributed-cost time-series.
  • concepts/data-contract — canonical wiki instance via Netflix FPD's producer-coordination primitive; every onboarded Netflix platform (Spark, etc.) agrees to schema + semantics + SLA before FPD ingests.
  • patterns/chargeback-cost-attribution — extended with the pre-chargeback: platform-data-layer attribution variant. Netflix FPD/CEA is upstream of the chargeback tier: it produces the attributed-cost time-series that any chargeback mechanism would consume.
  • concepts/capacity-efficiency (Meta framing) — adjacent program axis; Netflix focuses on upstream data correctness and transparent attribution while Meta focuses on offense/defense/AI-agent optimisation loops above such a substrate.

Recent articles

  • 2026-04-06 — sources/2026-04-06-netflix-stop-answering-the-same-question-twice-interval-aware-caching-for-druid (Ben Sykes' Netflix Performance Engineering post on an experimental interval-aware caching layer in front of Apache Druid for rolling-window dashboards. Netflix runs >10 trillion rows in Druid at 15M events/sec ingest; one popular dashboard generates ~192 queries/sec (26 charts × 64 queries × 30 viewers / 10-second refresh) mostly for near-identical data. Druid's full-result cache misses on every window shift + refuses to cache realtime-segment results. The new layer decomposes queries into granularity-aligned time buckets (1-min minimum) keyed as map-of-maps — SHA-256 query-shape hash outer key + big-endian timestamp inner keys for lex-order range scans. Per-bucket age-based TTLs (5 s for <2-min-old → 1-hour cap, doubling per additional minute of age) handle late-arriving data without a uniform-TTL trade-off. Contiguous- prefix lookup with one narrowed Druid fetch for the missing tail (patterns/partial-cache-hit-with-tail-fetch); negative caching for interior empty buckets but not trailing empty buckets. Intercepting-proxy deployment at the Druid Router = zero client changes. Storage on KVDAL / Cassandra — first wiki-documented KVDAL consumer use case beyond the launch post, exercising per-inner-key TTLs + inner-key range scans. Production: 82% queries get ≥partial hit, 84% result data from cache, P90 ~5.5 ms; A/B: ~33% drop in Druid queries, ~66% P90 improvement, up to 14× result- bytes reduction. Declared experimental; long-term upstream into Druid Brokers natively. The patterns/interval-aware-query-cache pattern is flagged as non-Druid-specific — applicable to any time-series DB with overlapping-window queries. Canonical concepts/staleness-vs-load-tradeoff framing on the wiki: 5 s TTL ≈ pipeline P90 ingestion lag → cache adds negligible staleness on top of what's already there.)
  • 2026-04-04 — sources/2026-04-04-netflix-powering-multimodal-intelligence-for-video-search (Netflix Search Engineering's architectural overview of the ingestion and fusion pipeline behind multimodal video search. Three decoupled stages: (1) transactional persistence of raw per-model annotations in Marken over Cassandra with "data integrity and high-speed write throughput" as the only job; (2) offline data fusion triggered by Kafka — discretizes continuous-interval annotations into fixed-size time buckets (worked example: one-second buckets), computes cross-model intersections (concepts/multimodal-annotation-intersection) like "Joey" character recognition × "kitchen" scene detection co-occurring at second 4; enriched records written back to Cassandra as "a highly optimized, second-by-second index of multi-modal intersections"; (3) indexing into Elasticsearch as nested documents (concepts/nested-document-indexing) keyed by (asset_id, time_bucket) for composite-key upsert (concepts/composite-key-upsert) idempotency across model re-runs. Nested shape enables "highly efficient, cross-annotation queries at scale" — find buckets where character and scene annotations co-occur. Sample annotation

  • intersection record JSON disclosed in post. Architecture density ~100% of the body; no scale numbers / latency percentiles / bucket-size disclosure / fusion-scheduling detail. First canonical wiki instance of patterns/three-stage-ingest-fusion-index + patterns/offline-fusion-via-event-bus + patterns/temporal-bucketed-intersection + patterns/nested-elasticsearch-for-multimodal-query — a reusable four-pattern stack for multimodal-temporal ingest. Extends Cassandra's wiki coverage with the dual-role substrate framing (raw + fused); extends Kafka with offline-fusion trigger bus role (sibling of Distributed Counter rollup-trigger); extends Elasticsearch with the nested documents for multimodal query role. Adjacent to MediaFM (2026-02-23 ingest) on the Netflix content-understanding axis but at a different altitude — MediaFM fuses per-shot multi-modal embeddings via a learned Transformer encoder; this pipeline fuses per-bucket annotations via a rule-based intersection. Fourteenth Netflix first-party ingest and first canonical multimodal-ingest-pipeline post on the wiki.)

  • 2026-04-02 — sources/2026-04-02-netflix-smarter-live-streaming-vbr-at-scale (Netflix Live Encoding + Live CDN (Renata Teixeira, Zhi Li, Reenal Mahajan, Wei Wei) document the 2026-01-26 fleet-wide cutover of all Netflix Live events from CBR to capped VBR (QVBR) on AWS Elemental MediaLive. Three-axis A/B wins at matched quality vs CBR: ≈5% fewer rebuffers per hour, ≈15% fewer bytes on average, ≈10% lower peak-minute traffic — the last is the Open Connect capacity-planning metric and a direct CDN provisioning win. Two structural problems Netflix had to fix before cutover: (1) VBR breaks current-traffic-as-capacity-proxy admission control — a stream currently emitting 2 Mbps of its 5 Mbps nominal fools steering logic into admitting more sessions, then the correlated spike on the next hard scene saturates the link. Fix: reserve capacity against nominal, not current (new canonical pattern). (2) "Same nominal" means different things under CBR vs VBR, so reusing the CBR ladder lost ≈1-VMAF-point on the bottom rungs. Fix: rung-by-rung VMAF-matched ladder tuning (new canonical pattern), bumping nominal only where the regression was > ≈1 VMAF point. End-to-end rollout playbook canonicalised as patterns/cbr-to-vbr-live-rollout. Forward work: feed upcoming-segment-sizes to device-side ABR algorithms; apply a measurement-informed "discount" on nominal- reservation to recover statistical-multiplexing headroom. Extends concepts/rebuffering-rate with a second canonical Netflix rebuffering-delta datum (first was AV1's 45% fewer vs AVC/HEVC; this is VBR's 5% fewer vs CBR at matched quality) and canonicalises VBR / CBR / capped VBR / QVBR / bitrate ladder on the wiki. Fifteenth Netflix first-party ingest and first canonical live-streaming rate-control migration post on the wiki.)

  • 2026-03-03 — sources/2026-03-03-netflix-optimizing-recommendation-systems-with-jdks-vector-api (Harshad Sane's Netflix Ranker serendipity scoring optimization retrospective. Video serendipity scoring — "how different is this candidate title from what you've been watching?" — consumed 7.5% of total CPU on every Ranker node. Five-step canary- validated optimization: (1) naive nested-loop O(M×N) cosine similarities → batched matrix multiply C = A×Bᵀ (patterns/batched-matmul-for-pairwise-similarity), (2) first cut regressed ~5% on double[M][D] per-batch allocations + GC pressure, (3) flat double[] row-major buffers + ThreadLocal<BufferHolder> grow-never-shrink reuse (patterns/flat-buffer-threadlocal-reuse) eliminated per- request allocation and restored cache locality, (4) tried netlib- java BLAS + native BLAS, lost in production to JNI transition overhead + F2J-vs-native confusion + row-vs-column-major translation costs alongside upstream TensorFlow allocations, (5) pure-Java SIMD via JDK Vector API with DoubleVector.SPECIES_PREFERRED + fma() per-lane FMA, scalar fallback via Lucene-inspired loop- unrolled dot product behind a MatMulFactory class-load probe (patterns/runtime-capability-dispatch-pure-java-simd). Production results on canaries confirmed at full rollout: ~7% drop in CPU utilization, ~12% drop in average latency, ~10% improvement in CPU/RPS, and the per-operator feature cost fell from 7.5% → ~1% of node CPU. Traffic-shape datum: ~98% single-video / ~2% large-batch requests, but ~50:50 by total video volume — batching was worth it for fleet cost even though it couldn't move p50. Canonical wiki lesson: "algorithmic improvements don't matter if the implementation details — memory layout, allocation strategy, and the compute kernel — work against you." 83 HN points. Thirteenth Netflix first-party ingest and first Netflix JVM-performance ingest after the 2024-07-29 virtual-threads post.)

  • 2026-02-23 — sources/2026-02-23-netflix-mediafm-the-multimodal-ai-foundation-for-media-understanding (Netflix's MediaFM — first tri-modal (audio + video + timed- text) foundation model at Netflix. BERT-style Transformer encoder over shot-level fused embeddings (up to 512 shots per title), pre-trained with Masked Shot Modeling (MSM) — mask 20% of shots, predict the original fused embedding at masked positions via cosine distance. Fused embedding per shot = concat of SeqCLIP (video) + wav2vec2 (audio) + OpenAI text-embedding-3-large (timed text), unit-normalised to 2304 dims, then projected to the Transformer hidden dim. Two special tokens prepended: learnable [CLS] + title-metadata-derived [GLOBAL]. Muon optimiser on hidden parameters (the switch is flagged as "noticeable improvements"), AdamW on the rest. Frozen encoder + per-task linear probes on five Netflix downstream tasks — ad relevancy, clip popularity ranking, clip tone, clip genre, clip retrieval — all beaten by MediaFM vs baselines. Ablation: contextualisation dominates over multimodality — uncontextualised tri-modal concat can actually hurt on clip popularity ranking; the transformer lifts it significantly above both flat baselines. Explicit inference rule "embedding in context" — run MediaFM on the full containing title and slice out the clip's shot span, not on the clip alone. Production consumers include cold start of newly-launching titles in recommendations — content-derived embedding ready for new content at launch, no user-interaction signal required. First content-embedding foundation model on the Netflix wiki axis — distinct from ML-platform / workflow-orchestrator / codec / observability / media-production / data-gateway / knowledge- graph axes. Introduces systems/netflix-mediafm + systems/netflix-seqclip + systems/wav2vec2 + systems/openai-text-embedding-3-large as systems; five new concepts concepts/masked-shot-modeling + concepts/shot-level-embedding + concepts/embedding-in-context + concepts/muon-optimizer

  • concepts/linear-probe-evaluation; two new patterns patterns/tri-modal-embedding-fusion + patterns/frozen-encoder-linear-probe. Netflix flags Qwen3-Omni / pre-trained multimodal LLMs as the likely future successor for the "fuse yourself" approach.)

  • 2026-01-02 — sources/2026-01-02-netflix-the-netflix-simian-army (Yury Izrailevsky + Ariel Tseitlin's canonical foundational post on chaos engineering at Netflix — originally published 2011, Medium-republished 2026-01-02, so oldest Netflix ingest on the wiki by original content date. Declares the Simian Army — eight named automated agents that continuously exercise Netflix's fault-tolerance design in AWS production: Chaos Monkey (random instance termination), Latency Monkey (RPC-boundary delay injection; large delays simulate outage without instance teardown), Conformity Monkey (operational-best-practice drift — terminates instances not in ASGs), Doctor Monkey (health-check + CPU-load unhealthy-instance detection with two-phase eviction), Janitor Monkey (unused- resource cleanup), Security Monkey ("extension of Conformity Monkey" — mis-configured AWS security groups + expiring SSL/DRM certs), systems/netflix-10-18-monkey|10-18 Monkey (l10n/i18n drift across geographies / languages / character sets), Chaos Gorilla (full AZ outage simulation — verifies automatic re-balance without user impact or manual intervention). Core claim: "just designing a fault tolerant architecture is not enough. We have to constantly test our ability to actually survive these 'once in a blue moon' failures." Business-hours induction under engineer supervision is deliberate: "By running Chaos Monkey in the middle of a business day, in a carefully monitored environment with engineers standing by to address any problems, we can still learn the lessons about the weaknesses of our system." Flat-tire analogy: practising in your driveway is "expensive and time-consuming in the real world, but can be (almost) free and automated in the cloud" — chaos engineering is a cloud-native discipline because the test itself is cheap. The post is partly aspirational — "much remains an aspiration — waiting for talented engineers to join the effort and make it a reality" — and is taxonomic in nature: names the family of failure-mode agents before any paper on chaos engineering as a discipline existed in the literature. No operational numbers, no code, no architecture diagram. Introduces systems/netflix-simian-army + all 8 monkey systems + concepts/chaos-engineering + concepts/random-instance-failure-injection + concepts/availability-zone-failure-drill + concepts/graceful-degradation + patterns/continuous-fault-injection-in-production + patterns/simian-army-shape — the foundational chaos-engineering vocabulary of the wiki. The vocabulary predates the 2016 coining of the term "chaos engineering"; Netflix built the practice before the discipline was named. Twelfth Netflix ingest on the wiki. 21 HN points on the Medium republication.)

  • 2025-07-29 — sources/2025-07-29-netflix-linux-performance-analysis-in-60-seconds (Netflix Performance Engineering's canonical 60-second Linux triage checklist — 10 stock shell commands (uptime, dmesg | tail, vmstat 1, mpstat -P ALL 1, pidstat 1, iostat -xz 1, free -m, sar -n DEV 1, sar -n TCP,ETCP 1, top) run in a defined order as the first response on any Linux host performance issue. Encodes Brendan Gregg's USE Method (Utilisation / Saturation / Errors) across CPU / memory / disk / network using only /proc-backed tools + the sysstat package. Canonical interpretation rules: vmstat's r > CPU count = CPU saturation; iostat's %util > 60% usually hurts + avgqu-sz > 1 = saturation (with LVM / virtual-disk caveat); %sys > 20% = kernel-inefficiency hint; %steal > 0 = hypervisor co-tenancy signature; free -m's -/+ buffers/cache row is the load- bearing memory accounting (ZFS ARC caveat). Worked examples from Titus-era prod hosts: load average 30 resolved to user-CPU-bound via r ≈ 32 on 32-CPU box; dmesg catching perl OOM-killer

  • TCP SYN-flood; two Java processes at 1591% + 1583% CPU in pidstat. Explicit handoff to deeper tools (eBPF, flame graphs, Atlas). First canonical USE-Method + first-response-checklist post on the wiki.)

  • 2025-07-03 — sources/2025-07-03-netflix-av1scale-film-grain-synthesis-the-awakening (Netflix Video Algorithms rolls out AV1 Film Grain Synthesis (FGS) at scale on the streaming service. FGS has been in the AV1 standard since inception but was only enabled on "a limited number of titles" at Netflix's 2021 AV1-on-TVs launch; this post documents the 2025-07 at-scale rollout. Architectural bet: denoise the source before compression, encode the clean signal, transmit AR coefficients + piecewise-linear scaling function as grain metadata, re-synthesize the grain on the decoder — block-based 32×32-patch tiling from a 64×64 noise template, cheap on commodity consumer devices. Compresses the worst-case block-transform input (near-random grain) by ejecting it from the bitstream entirely. Netflix reports "significant bitrate savings" on grain-heavy titles (e.g. They Cloned Tyrone), preserving artistic intent. Introduces systems/av1-codec + concepts/film-grain-synthesis + concepts/auto-regressive-grain-model + concepts/grain-intensity-scaling-function + concepts/denoise-encode-synthesize to the wiki, plus the generalised patterns patterns/decoder-side-synthesis-for-compression + patterns/codec-feature-gradual-rollout. Extends concepts/video-transcoding with the synthesis-based encoding-pipeline shape and concepts/visual-quality-metric with the why reference metrics break down on synthesis-based codec tools methodology gap. The AV1 standard does not specify the encoder-side denoiser — that's where per-vendor investment lands, and where Netflix's 2021 → 2025 gap went. 255 HN points.)

  • 2025-06-14 — sources/2025-06-14-netflix-model-once-represent-everywhere-uda (Netflix Content Engineering introduces UDA (Unified Data Architecture) — an in-house knowledge-graph platform that unifies data catalog + schema registry with a hard semantic-integration requirement. Core thesis: "define a model once, at the conceptual level, and reuse those definitions everywhere … the conceptual model must become part of the control plane" (patterns/model-once-represent-everywhere). Business concepts (actor, movie, asset) and system domains (GraphQL, Avro, Data Mesh, Mappings) are authored as domain models in Upper, UDA's metamodel — a bootstrapping upper ontology that is self-referencing / self-describing / self-validating (patterns/self-referencing-metamodel-bootstrap). Built on a strict subset of W3C semantic tech (RDF + RDFS

  • OWL + SHACL) with enumerated gaps: RDF lacked a usable info-model over named graphs; SHACL's global-URI + single-data-graph assumptions don't fit enterprise local-keys; ontology tooling lacked GraphQL-Federation-style modular collaborative authoring; teams lacked shared authoring practice. UDA's response: a named-graph-first info model where every named graph conforms to a governing named graph, all the way up to Upper. All domain models are conservative extensions of Upper — the formal composition rule that keeps semantic integration stable as domains accumulate. Transpiler family (patterns/schema-transpilation-from-domain-model) projects each domain model into GraphQL / Avro / SQL / RDF / Java — schemas + auto-provisioned data-movement pipelines (federated GraphQL → Data Mesh, CDC → Iceberg data products) are generated together from the same source. Upper's own projection (Jena-based Java API + federated GraphQL schema) lands in Netflix's Enterprise GraphQL Gateway — the metamodel is its own first customer. Two named production consumers: PDM (Primary Data Management) — flat/hierarchical taxonomies + generated authoring UI; Avro schemas auto-provisioning warehouse data products; GraphQL schemas auto-provisioning Enterprise Gateway APIs; canonical wiki instance of model-once-represent-everywhere applied to reference data. Sphere — self-service operational reporting; user picks concepts in familiar business vocabulary, Sphere walks the knowledge graph and generates SQL against the warehouse (patterns/graph-walk-sql-generation); knowledge graph is the planner, warehouse is the executor. Three introspection surfaces: Java (generated from Upper) / federated GraphQL / SPARQL — model-once applied to the runtime API. Open repo + onepiece.ttl worked example at github.com/Netflix-Skunkworks/uda. Caveats: architecture- overview voice only — no scale / adoption / transpiler numbers; mappings domain model underspecified; named-graph resolution mechanism unnamed; governance/ownership primitives gestured at not defined; first post in a series with information- infrastructure detail deferred. Second wiki knowledge-graph framing (enterprise data-integration substrate) alongside the existing Dropbox-Dash agent-retrieval-substrate framing — different loads, same data structure. Seventh Netflix ingest.)

  • 2025-04-08 — sources/2025-04-08-netflix-how-netflix-accurately-attributes-ebpf-flow-logs (Netflix rebuilds its eBPF flow-log attribution pipeline, replacing a Sonar discrete-event IP-assignment stream with a heartbeat-based architecture. Under the old system, ~40% of Zuul's reported downstream dependencies were misattributed; under the new design, a 2-week validation window showed zero misattribution. Architectural split: local IPs resolved in-kernel at capture time via either a Metatron cert on EC2 or an IPMan-populated eBPF map on Titus containers (with a second (IPv4, port) → workload map disambiguating Netflix's NAT64-free IPv6-to-IPv4 translated sockets); remote IPs resolved at FlowCollector against a per-IP list of non-overlapping (workload, t_start, t_end) time ranges populated entirely from flow heartbeats and broadcast to peer nodes via Kafka. Amazon Time Sync's sub-ms clock accuracy makes wall-clock time ranges a reliable attribution key (concepts/amazon-time-sync-attribution). Per-region partitioning + a CIDR trie over VPC CIDRs for cross-region flow forwarding (patterns/regional-forwarding-on-cidr-trie) avoids global broadcast (only ~1% of flows are cross-regional). Sonar is retained only for AWS ELB IPs where heartbeat-based attribution is impossible. 1-minute disk buffer for remote attribution replaces the old 15-minute holdback. Operating footprint: 30 c7i.2xlarge instances at 5M flows/sec with no persistent storage — in-memory state rebuilt from incoming heartbeats on cold start. Explicit correctness-over-coverage design posture: "a small percentage unattributed is acceptable, any misattribution is not." Canonical wiki instance of concepts/discrete-event-vs-heartbeat-attribution as a distributed-systems primitive; extends the existing Netflix eBPF

  • Titus + Atlas corpus with a second major Titus-resident eBPF system alongside the 2024-09-11 noisy-neighbor run-queue-latency monitor.)
  • 2025-04-01 — sources/2025-04-01-netflix-globalizing-productions-with-netflixs-media-production-suite (Netflix's Media Production Suite (MPS) inside Content Hub. Cloud-based filmmaker toolchain covering the production lifecycle; >350 titles across UCAN / EMEA / SEA / LATAM / APAC have used ≥1 MPS tool. Seven named tools: Footage Ingest (drive → cloud gateway), Media Library, Dailies, Remote Workstations, VFX Pulls, Conform Pulls, Media Downloader. ~200 TB OCF / title average; outliers up to ~700 TB. Hybrid-cloud infrastructure: AWS as durable substrate; Open Connect as high-bandwidth ingest-centre ↔ AWS backbone (first non-streaming Open Connect role on wiki); regional ingest centres rolling out globally where drives are dropped off + uploaded "within a matter of hours." Footage Ingest pipeline stages: validate manifest → upload OCF + OSF → checksum validation → inspect + metadata extraction → build playable proxies → tier-2 cloud archive. LTO tape creation default-off — "when utilizing MPS, we don't require LTO tapes to be written unless there are title-specific needs." Standards-driven automation thesis: ACES + AMF (colour), ASC MHL (checksum / manifest), ASC FDL (framing interoperability), OTIO (timeline interchange) — open standards make automation O(producers + consumers) instead of O(producers × consumers) and "offer high-complexity workflows to markets or shows that don't normally have access to them." Fuzzy-metadata EDL → OCF matching in production for VFX Pulls + Conform Pulls; perceptual-CV conform under investigation. Centralised cloud library replaces per-vendor I/O surfaces with Content Hub Workspaces (Google-Drive-style shared folders). Remote-monitoring dashboard over Footage Ingest activity stream replaces out-of- band phone-call status checks. Worked example: Brazilian F1 series Senna (2023) with editorial in Porto Alegre + Spain and VFX across Brazil / Canada / US / India via Scanline VFX — cross-country production shipped without hand-carried drives. Caveats: announcement voice; ingest-centre count + per-centre bandwidth not disclosed; partial-upload + checksum-mismatch semantics undescribed; tier-2 archive storage class undisclosed; perceptual-match model architecture + accuracy undisclosed.)
  • 2024-11-13 — sources/2024-11-13-netflix-netflixs-distributed-counter-abstraction (Netflix's Distributed Counter Abstraction — the third mature abstraction on the Data Gateway platform (after KV DAL and TimeSeries). ~75K count req/s globally at single-digit-ms latency. AtomicInteger-shaped API with IdempotencyToken (event_time, nonce). Two-mode taxonomy: Best-Effort is a thin wrapper over EVCache incr/decr (no cross-region, no consistency, no idempotency — retry-unsafe); Eventually Consistent is the load-bearing event-log + background sliding-window rollup design. Each mutation persisted as an event in TimeSeries (Cassandra-backed) with composite (event_time, event_id, event_item_key) natural idempotency key + bucketed time partitioning. Background rollup pipeline: light-weight {namespace, counter} events go to in-memory per-instance queues, XXHash-routed, Set-coalesced per rollup window; batches query TimeSeries in parallel within an immutable aggregation window governed by TimeSeries acceptLimit; new checkpoint (lastRollupCount, lastRollupTs) lands in Cassandra Rollup Store
  • EVCache Rollup Cache; reads serve the cache as a point-read + trigger a rollup for self-healing. Adaptive back-pressure on rollup batches; last-write-timestamp via Cassandra USING TIMESTAMP is the drain-vs-circulate discriminator for low-vs-high-cardinality counters. Experimental Accurate mode computes lastRollupCount + delta(lastRollupTs, now()) in the read path. Named future work: regional rollup tables + global reconciliation to handle cross-region replication drift; durable rollup queues + handoffs for infrequently-accessed counters. Canonical wiki instance of event-log-based counters, immutable aggregation windows, sliding-window rollup aggregation, light- weight rollup events, and fire-and-forget rollup triggers. Rejected alternatives (single-row + CAS, per-instance aggregation, durable queue + stream processor, raw event log) each walked through with named failure modes; HyperLogLog + Count-Min Sketch named and rejected.)
  • 2024-09-19 — sources/2024-09-19-netflix-netflixs-key-value-data-abstraction-layer (Netflix's KV DAL — the most mature abstraction on the Data Gateway platform. gRPC service in front of Cassandra + EVCache + DynamoDB + RocksDB; two-level-map data model; namespace-routed; client-generated (generation_time, nonce) idempotency tokens with sub-millisecond EC2 Nitro clock skew enabling hedged + retried writes on last-write-wins stores; transparent chunking of items

    1 MiB with one token binding chunk writes atomically; client-side payload compression (75% reduction in Netflix Search); byte-size pagination + adaptive pagination + SLO-aware early response for predictable single-digit-ms page-read latency; in-band signaling handshake propagating target/max SLOs. TTL-jitter deletes to avoid compaction load spikes; single-tombstone deletes for record + range scope. Named production consumers: streaming metadata, user profiles, Pushy (push-messaging registry), Bulldozer (impression persistence).)

  • 2025-01-02 — sources/2025-01-02-netflix-cloud-efficiency-at-netflix (Program-level overview of Netflix's internal cloud-efficiency data platform. Two-layer design: FPD (Foundational Platform Data) normalises inventory/ownership/usage per platform via data contracts with producers; CEA (Cloud Efficiency Analytics) applies per-platform business logic over FPD to produce attributed-cost time-series with single-owner resolution + multi-tenant distribution + multi-aggregation output. Published SLAs; transparent compartmentalised model so consumers can trace how a dollar was attributed. Three named program tensions — "A Few Sizes to Fit the Majority", "Data Guarantees", "Abstraction Layers". Forward direction: extend FPD beyond cost into security/availability; move CEA from descriptive to predictive-anomaly-detection. No raw numbers disclosed. First Netflix canonical post on the cost-attribution + capacity-efficiency axes.)
  • 2024-09-11 — sources/2024-09-11-netflix-noisy-neighbor-detection-with-ebpf (Per-container run-queue-latency monitor on eBPF running on Titus fleet. tp_btf/sched_wakeup + tp_btf/sched_switch tracepoints; PID-keyed hash map computes runq_lat in-kernel; cgroup_id derived via BPF RCU kfuncs; in- kernel per-cgroup-per-CPU rate limiter before ringbuf; Go agent emits Atlas percentile timer + preempt-cause-tagged counter. Baseline p99 ≈ 83.4 µs; elevated runq.latency alone is ambiguous between noisy neighbor and self CFS-quota throttling — resolved by pairing with sched.switch.out tagged by preempting cgroup class. Canonical scheduler-layer instance of concepts/noisy-neighbor; introduces 3 new patterns and 3 new concepts to the wiki; stub pages for systems/netflix-atlas + systems/netflix-runq-monitor.)
  • 2024-07-22 — sources/2024-07-22-netflix-maestro-netflixs-workflow-orchestrator (Maestro open-sourcing + architectural deep dive. Horizontally scalable single-cluster orchestrator: ~500K jobs/day avg / ~2M peak / 87.5% YoY; acyclic + cyclic workflows with engine-native foreach + subworkflow + conditional-branch primitives; five named run strategies — Sequential / Strict-Sequential / First-only / Last-only / Parallel-with-Concurrency-Limit; homemade SEL — JLS-subset safe expression language with loop / array / memory runtime limits + Java Security Manager sandbox — for safe code injection in parameterized workflows; seven-layer step parameter merging pipeline; signal-based step dependencies with exactly-once guarantee + signal lineage; per-step breakpoints for IDE-style workflow debugging + in-flight state mutation; platform-vs-user retry distinction; eventually-consistent rollup model across nested subworkflows + foreach; two-tier internal → external event publishing (SNS / Kafka).)
  • 2024-07-22 — sources/2024-07-22-netflix-supporting-diverse-ml-systems-at-netflix (MLP / Metaflow integration with Titus, Maestro, Fast Data, Cache, Hosting, Amber; foundational-platform + domain-libraries thesis; hundreds of Metaflow projects in prod; 260M+ subscribers / 190+ countries via Content Decision Making; Explainer flow as dynamic-environment-composition instance; Amber on-demand feature compute via Hosting queues.)

Ingest posture

Netflix is a Tier-1 source — ingest eagerly; the TechBlog is cross-referenced widely (eBPF, container platforms, chaos engineering, video codecs, ML platform, storage). Filter for product-launch / culture / hiring posts; everything architectural belongs on the wiki.

Last updated · 319 distilled / 1,201 read