Skip to content

META 2026-10-27

Read original ↗

Meta — PrivacyLib: Privacy Controls at the Data Access Boundary

Summary

A 2026-10-27 Meta Engineering deep dive into PrivacyLib, the enforcement-and-monitoring layer of Meta's Privacy Aware Infrastructure (PAI). Historically every new privacy requirement (purpose limitation, data-flow discovery, …) was implemented separately inside the client of every data and processing system, so a cross-system rollout took months. PrivacyLib separates system integration from requirement implementation: each system integrates once by mapping its native APIs onto five common operations (read, write, RPC, request entry, inline execution), and each requirement ships as a reusable module — enforcing modules (e.g. Policy Zones for purpose limitation) can block an access, observing modules (e.g. Privacy Probes for lineage discovery, Telemetry Logging, Agent Observability) only record it. PrivacyLib runs in-process inside the clients of hundreds of systems — >100 client libraries, thousands of services, millions of assets/day, billions of checks per second — because the client still holds caller identity, endpoint and purpose that the datastore would lose. It stays cheap via build-time compilation of requirements, per-request and per-host caches (the earlier remote-config design was thousands of times slower), and Bloom-filter telemetry downsampling. Raw RPCs that bypass integrated clients are backstopped at ServiceRouter via a cryptographically signed token from the client-side check, with a coarser, strictest-interpretation fallback when the token is absent. Changes roll out safely through two paths: annotation edits are simulated against observed data flows during code review and then ship through each service's own deploy pipeline (bounding blast radius), while new modules are gated by a rollingOut() runtime percentage moved 0→100%. The same foundation now governs internal AI agents: AgentLib attaches agent-session context to tool calls, propagated to the PrivacyLib data-access boundary so Policy Zones can block disallowed access even for tools the agent's developer didn't anticipate.

Key takeaways

  1. Integrate once, ship requirements as modules. Before PrivacyLib, "implementing a new requirement across systems could take months" because enforcement was re-implemented per system per requirement. Now a system integrates once and "a new requirement no longer needs a separate implementation in each system"; enabling an additional module on an already-integrated system "takes only days". Agent Observability was picked up by every integrated system "without new integration work" — "each new privacy capability arrives as a module, not as a project." (Source: this article)
  2. Enforcing vs observing modules. Enforcing modules (Policy Zones) can deny an access; observing modules (Privacy Probes, Telemetry Logging, Agent Observability) only record. PrivacyLib "combines the results from enforcing modules into one response"; observers "emit telemetry without affecting the response". Read/write/RPC have post-operation variants so observers run after the decision. (Source: this article)
  3. A small common operation model is the integration contract. Every integration — KV stores, graph DBs, ML pipelines, filesystems, Logger — maps its access points to five operations: Read, Write, RPC, Request entry, Inline execution (wraps a computation like a feature evaluation with its own privacy context). Modules are written against these five, not against native APIs. (Source: this article)
  4. Enforce at the client, not the datastore. In-process evaluation keeps "richer context" (caller identity, endpoint, purpose — "frequently lost if the check is evaluated on the datastore") and gives failure isolation: "A faulty change is contained to the service running that version of the code, keeping the shared datastore outside the failure boundary." (Source: this article; concepts/blast-radius)
  5. Annotations live with the data destination, and are code-generated into a common artifact. A Logger destination's LoggerConfig carries the MESSAGING_TRAFFIC_METADATA annotation once — not per writer, which could "disagree, or a new writer could omit it entirely". Logger's codegen pipeline translates it into a PrivacyLib artifact (Hack class directly; for C++/Python, Logger publishes to a central catalog from which PrivacyLib generates language-specific artifacts). (Source: this article; concepts/policy-as-data)
  6. Make the per-check cost nearly zero. Requirements are "compiled at build time into their simplest evaluable form"; "most checks" are answered from per-request and per-host in-process caches. The earlier remote-config-fetch design was "thousands of times slower than the in-process path". Fleet-wide cost: "only a small fraction of Meta's total compute capacity." (Source: this article)
  7. Telemetry must be deduplicated, not exhaustive. PrivacyLib records module status, samples latency, logs errors; per-host Bloom filters downsample high-volume repeated patterns at a fixed rate while fine-grained sampling rates adapt to traffic — "deduplicate repeated events while keeping rare ones visible." (Source: this article)
  8. Two-layer coverage: rich client check + signed token + coarse RPC backstop. A successful client-side check attaches a cryptographically signed token; ServiceRouter validates it on supported RPC traffic. Tokenless requests get a coarser payload-only check that "applies the strictest interpretation, choosing false positives over gaps" — a fail-closed backstop in a layered design. Paths outside both layers still need their own integration. (Source: this article)
  9. Two change paths, two safeguards. (a) Annotation changes (a two-line LoggerConfig edit that can block many writers at once): during code review the regenerated artifact and the old one are replayed against observed data flows; any flow whose decision changes is reported, and the source edit + generated artifact + simulation results are reviewed together. Then the artifact ships via each service's own deploy pipeline, not a fleet-wide config push. Trade-off: slower propagation, handled by explicit tracking to deadlines plus emergency deploys. (b) Module rollouts (affect every destination on a path): rollingOut() puts execution behind a runtime control keyed e.g. logger.input.write.zones, moved 0→100% while watching denials, errors, latency, and lowered "without rebuilding or redeploying"; then flipped to enabled(). (Source: this article; patterns/staged-rollout, patterns/progressive-configuration-rollout, concepts/feature-flag)
  10. Agents: enforce on what they attempt, not what was expected. Agent data paths are determined at runtime. AgentLib associates each tool call with its agent session; downstream clients propagate that context to the PrivacyLib boundary, where Agent Observability records session/tool-call IDs + destination + client API + integration point + errors (queryable per session), and the unmodified Policy Zones module enforces using the agent context. Blocks propagate back up the call chain. "The access decision does not depend on the agent behaving as intended." (Source: this article)
  11. Reliability earns adoption — learned from incidents. "Early on, an invalid config crashed processes across the fleet"; other incidents came from annotation changes that "compiled successfully but changed access decisions and caused product failures." Service owners only trust a hot-path library if they can see a change's effect before release and recover "without a fleet-wide rollback". And "coverage includes deployment": older services must redeploy to pick up new capabilities, so inventory + tracking is part of coverage. (Source: this article)

Walkthrough: protecting messaging metadata

Messenger backend services write non-content messaging metadata (send/deliver timing etc.) to warehouse tables via Logger, an in-process client library. Flow:

  1. MetadataTraceLoggerConfig uses MessagingTrafficMetadataPolicyTrait and calls annotateWithTableLevelPolicy() → destination carries MESSAGING_TRAFFIC_METADATA.
  2. Codegen emits MetadataTraceLoggerConfigPrivacyLibArtifact (asset ID + purpose-policy vars).
  3. Logger's Hack integration declares once for all destinations: PrivacyLibOperation::write() with enforcements zones(ZoneDataType::LOGGER)->withPropagation()->enabled() and telemetryLogging()->enabled(). Adding a module here needs no change to destinations or call sites.
  4. In C++, Logger builds a PrivacyLibBuilder (client name, asset list = config name, hashed API enum mapping log → write, Caller::LOGGER, the write implementation) and calls dispatch().
  5. Policy Zones checks (i) current context is an allowed use of messaging metadata, (ii) destination artifact carries the annotation. Either fails → deny before Logger sends the record downstream.

Operational numbers

Metric Value
Client libraries integrated >100 (KV stores → AI features)
Production services with the library thousands
Assets accessed daily millions
Checks billions per second
Earlier remote-config fetch vs in-process path thousands of times slower
Time to enable a module on an integrated system days (vs months bespoke)
Fleet compute share "small fraction" (undisclosed)
Common operations 5 (read, write, RPC, request entry, inline execution)

Systems / concepts / patterns extracted

New page: - systems/meta-privacylib — primary subject. AgentLib and the Agent Observability module are documented as sections on it (Systems gate rule 2).

Enriched: - systems/meta-policy-zones — as the canonical PrivacyLib enforcing module; Logger write path; agent enforcement. - systems/meta-privacy-aware-infrastructure — PrivacyLib as the final "continuously enforce and monitor" stage. - systems/servicerouter — lower-boundary signed-token validation + strictest-interpretation fallback. - concepts/purpose-limitation (new — now 2 sources), concepts/information-flow-control (new — now 2 sources), concepts/data-lineage, concepts/bloom-filter, concepts/blast-radius, concepts/defense-in-depth, concepts/fail-open-vs-fail-closed, concepts/policy-as-data, concepts/feature-flag. - patterns/staged-rollout, patterns/progressive-configuration-rollout.

Listed only (Systems gate — no page): meta-agentlib (section on PrivacyLib), meta-privacy-probes (named observing module; deep-dived in a 2025 Meta lineage post not yet ingested), meta-logger (Meta's logging client library).

Also enriched: systems/messenger — messaging-metadata worked example.

Tag-only (taxonomy gate): change-impact simulation of annotation edits, signed context token, enforcing-vs-observing modules, build-time policy compilation, agent-context propagation.

Caveats

  • No absolute per-check latency, cache hit rates, or compute share disclosed — only relative claims ("thousands of times slower", "small fraction").
  • Coverage is explicitly bounded: Meta-managed clients + supported RPC paths; anything else needs its own integration. Agent context propagation only on "supported paths".
  • The token's cryptographic scheme, lifetime and replay protection are not described.
  • First-party post; incident descriptions are brief (no COE-level detail).
  • Per-service deploy propagation is slower than a global config push — an accepted trade-off for blast-radius containment, not eliminated.

Source

Last updated · 854 distilled / 2,448 read