Skip to content

META 2026-09-02 Tier 1

Read original ↗

Meta — An Organizational Second Brain: Building an AI That Learns From Experts

Summary

Meta describes a domain-expert AI agent for a specific compliance domain that acts as a "secondary expert" — making specialist knowledge available to anyone in the org and preserving it durably. Its novelty is the integration of two layers: a structured, auditable knowledge architecture that separates what the agent knows from how it reasons, and a self-improvement loop that compiles expert feedback into verified, regression-tested updates without model retraining. Knowledge lives in 200+ document files in a strict taxonomy (positions, taxonomy/vocabulary, routing indexes, gateways) with depends_on / referenced_by YAML frontmatter forming a bidirectional dependency graph; reasoning lives in composable "recipes" (imperative, multi-step procedures that reference knowledge but contain no facts). The system explicitly cites Karpathy's LLM Wiki and Google's Open Knowledge Format as convergent ideas: knowledge should be pre-extracted, explicitly structured, and progressively disclosed rather than re-derived on every query. Every expert correction is treated as a compilation problem — diagnosed to a root cause, compiled into minimal edits, validated against blind replay + regression tests (with an independent adversarial reviewer and a deterministic linter), reviewed by a human, and then folded back into the regression suite so the gain is permanent. Results after ~6 weeks / three sprints: outputs rated useful almost all the time, assessment time days → minutes, zero regressions across improvement cycles.

Key takeaways

  1. Separate what the agent knows from how it reasons. Declarative knowledge files state positions but prescribe no procedures; imperative recipes prescribe procedures but contain no domain facts. This makes failure attribution tractable — a wrong answer is either a knowledge gap or a recipe flaw — and lets each layer evolve independently: adding a position is a new knowledge file + routing-index update with no recipe change; fixing methodology is a recipe edit with no knowledge change. (Source: this article) — see knowledge-reasoning-separation.

  2. Knowledge as a navigable filesystem with a bidirectional dependency graph. 200+ files in a strict taxonomy: Position files (authoritative org stances + constraints + machine-actionable routing implications), Taxonomy/vocabulary files (single-source-of-truth glossary), Routing indexes (map input characteristics → applicable positions/ procedures, making retrieval deterministic and auditable rather than embedding-similarity-only), and Gateway files (threshold tests the agent must pass before entering an analytical domain). Every file declares depends_on and referenced_by in YAML frontmatter — so when one file changes you can trace exactly what else is affected, which is what makes automated editing safe. (Source: this article) — see bidirectional-dependency-graph-frontmatter.

  3. Partition knowledge between wiki and RAG by density × usage frequency. High-density, frequently-referenced sources (positions, decision frameworks, boundary examples, strategic interpretations) go into the curated wiki — consulted on nearly every turn, kept current, versioned, validated. Sparse, situationally-relevant sources (detailed reference material, individual product specs, historical decision records, niche external knowledge) are served via semantic/lexical RAG — "loading all of them into the wiki would bloat the system and dilute attention." (Source: this article) — see density-frequency-knowledge-partition.

  4. Composable recipes + progressive disclosure cut per-turn tokens ~80%. Recipes compose into pipelines like a head chef's master recipe delegating to sub-recipes (sauce, protein, garnish). A top-level routing recipe examines the input and selects downstream recipes, each handling one analytical phase and carrying only the instructions + knowledge relevant to that phase. Early versions used a single flat instruction file and loaded all sources via semantic search on every run; after restructuring into recipe-driven stages, each query touches only a small targeted subset, cutting tokens per turn by ~80%. "Context windows are finite and attention degrades with volume." (Source: this article) — see composable-recipes, concepts/progressive-capability-disclosure.

  5. Humans stay in control via checkpoints and escalations. Checkpoints are defined points where the agent surfaces intermediate reasoning for expert confirm/correct/redirect before proceeding. Escalations fire on genuine ambiguity (underspecified input, or evidence supporting more than one defensible reading) — the agent hands the question to the expert rather than forcing a resolution. These serve three purposes at once: quality/direction control, training signal for the improvement loop, and trust calibration (experts see reasoning + uncertainty flags, not just final output). Recommended by default in compliance, financial risk, security review, and engineering safety. (Source: this article) — see checkpoint-and-escalation.

  6. Treat knowledge maintenance as a compilation problem. The self-improvement flywheel (called the most distinctive part of the system) moves every expert correction through four phases: (1) Diagnose feedback into actionable issues + root cause; (2) Compile issues into minimal verified edits; (3) Validate the fix works without regressions; (4) Human review. On completion, the fixed scenario is added to the regression suite so future changes must preserve it — "every fix permanently raises the bar." (Source: this article) — see self-improving-knowledge-base-loop, feedback-to-verified-edit-compilation.

  7. The attribution test beats classifying feedback by conversational form. Meta's first approach classified feedback by form (expert provided info → knowledge gap; expert redirected → procedure problem); this failed because conversational form is a poor proxy for root cause. The working approach separates extraction from classification: extract every substantive signal plus the agent's full knowledge manifest (every file loaded, when, how used), then apply one test — Could the agent have reached the correct conclusion from its source materials? Materials had the answer but agent erred → recipe problem; materials lacked the answer → knowledge gap; experts disagree → ambiguity, flagged for human discussion. (Source: this article) — see attribution-test-for-feedback.

  8. Two trust mechanisms in compilation: adversarial review + a deterministic linter. The compiler translates each diagnosed issue into minimal edits, with sub-agents analyzing impact in parallel (cross-references, conflicts, token-budget impact, test coverage, duplication). Independent adversarial review: a separate agent in a fresh context with no knowledge of the improvement rationale receives only the proposed diffs and hunts for contradictions/broken edge cases/undermined positions — it cannot inherit the proposers' blind spots. Deterministic structural validation: a linter programmatically catches dangling cross-references, file-size budget violations, identifier collisions, and dependency cycles — "this layer is not probabilistic. It passes or fails." (Source: this article) — see independent-adversarial-review, deterministic-structural-linter.

  9. Blind two-stage evaluation gates every change. Targeted replay runs the agent on the original triggering scenario — the agent does not know it is being tested, and a separate judge evaluates the new output against the original expert feedback without knowing what changed (deliberately blind → prevents confirmation bias). Regression testing runs domain benchmarks (structured Q&A suites; an independent LLM judge scores pass/fail where multiple correct answers exist) in parallel independent sessions. Failures at either stage retry compilation — regression failures retry with an updated prompt describing where the agent regressed. (Source: this article) — see blind-targeted-replay, regression-suite-enrichment-loop.

  10. Keep the complexity in text, not weights. "Every improvement is a text edit that a domain expert can review in 30 seconds. Every change is version-controlled, diffable, and reversible." The output of the pipeline is a pull request with a complete audit trail — a human reviews a proven fix rather than debugging a raw failure. The pattern generalizes to any domain governed by retrievable text rather than model weights: regulatory compliance, protocol adherence, financial risk assessment, security review, engineering standards compliance, procurement evaluation. (Source: this article)

Architecture (four interdependent layers)

The system has four layers, each solving a distinct problem, each depending on the others ("remove any one layer and the others degrade"):

  1. Knowledge system — the "organizational second brain." A long-running offline process reasons through source documents and distills them into structured knowledge files (curated statements of how the org interprets its domain, with constraints, boundaries, and machine-readable routing implications). File structure is what makes automated editing possible.
  2. Reasoning layer — composable recipes. Explicit procedures make failure attribution tractable.
  3. Evaluation framework — gates every change (blind replay + regression).
  4. Improvement loop — feeds corrections back into both knowledge and reasoning.

The self-improvement flywheel

expert correction
   → Diagnose (extract signals + knowledge manifest; attribution test → root cause)
   → Compile (sub-agents produce minimal edits; adversarial review; deterministic linter)
   → Validate (blind targeted replay + regression benchmarks; retry compile on fail)
   → Human review (PR with audit trail)
   → Land + enrich regression suite (fixed scenario becomes a permanent test)

Operational numbers

  • 200+ knowledge files in a strict four-part taxonomy.
  • ~80% reduction in tokens consumed per turn after moving from a single flat instruction file + load-all-via-semantic-search to recipe-driven progressive disclosure.
  • ~6 weeks / three development sprints to reach the reported results.
  • Days → minutes reduction in individual assessment time.
  • Zero regressions across improvement cycles (every fix strengthens the regression suite).
  • Domain SMEs rated outputs useful almost all the time; the agent handles the vast majority of the analytical work, leaving experts the genuinely ambiguous cases.
  • Attribution-test compiler validated via targeted replay + multi-benchmark regression with an independent LLM judge for multi-answer domains.

Requirements to adopt (Meta's checklist)

  1. A structured knowledge system with explicit file boundaries, cross-references, and a dependency graph.
  2. A procedural layer separating domain knowledge from analytical methodology (recipes).
  3. An automated evaluation suite that grows with each improvement cycle.
  4. Human-in-the-loop checkpoints calibrated to the domain's risk tolerance.

Caveats

  • Architecture-and-results voice: no model/LLM identity, no fleet size, no file-count breakdown per taxonomy category, no exact benchmark scores or numeric usefulness metric (only "useful almost all the time"), and the specific compliance domain is not named.
  • Results are for one domain after ~6 weeks; generalization to finance / security / engineering is asserted by structural argument, not yet measured across domains in the post.
  • The self-improvement loop's automation depends on the dependency graph being accurate; the post does not quantify how often the deterministic linter or adversarial reviewer catches a bad automated edit.

This wiki's own pattern

Meta's system is a production instance of the exact pattern this wiki follows (Karpathy's LLM Wiki, cited directly in the article): compile knowledge into persistent, interlinked files with dependency-declaring frontmatter, disclose it progressively, and maintain it with critic/lint passes rather than RAG re-derivation. It is the closest sibling on the wiki to Meta's own AI Pre-Compute Engine (offline multi-agent extraction of compass-not-encyclopedia context files) and Spotify's Vedder context layer, extended with a rigorous feedback→verified-edit compilation loop.

Source

Last updated · 766 distilled / 2,225 read