PATTERN Cited by 2 sources

Progressive data loading

What it is

An evaluation optimization for systems that combine (a) a declarative expression language with statically extractable data dependencies and (b) an engine that owns data fetching (concepts/data-policy-separation): load the declared data dependencies in batches, evaluate the expression after each batch under three-valued logic, and exit early as soon as the result is determinable — even if not all dependencies have been loaded.

Algorithm

partitionedLoadPlan = partition(requiredDependencies) by heuristic

loaded = {}                                  // initially empty
for batch in partitionedLoadPlan:
  loaded ∪= DatabaseLoader.load(batch)
  verdict = evaluate(expression, loaded)     // true / false / null
  if verdict != null:                        // enough info
    return verdict
return default                               // still unknown after all batches

Partition heuristics come from production data. Figma writes: "File, folder, and team roles are the second most common way (after link access) by which users gain access to resources; we prioritized these in our load plan." — the high-coverage dependencies are loaded first so the common case short-circuits before the rare-dependency batches run.

Why it matters

• Database round-trips avoided. Each skipped batch is one (or more) queries avoided. Figma reports this "more than halved the total execution time of our permissions evaluation" — i.e. more than 2× end-to-end speedup (Source: sources/2026-04-21-figma-how-we-built-a-custom-permissions-dsl).
• Load-balancing side benefit. The database sees less load, and the hot paths of the authorization system stop dominating aggregate database IOPS (Figma pre-DSL: "permissions checks were around 20% of the database load").
• Graceful degradation. If a later-batch data source is slow or down, many evaluations still complete successfully on the early-batch data.

Prerequisites

• Data–policy separation — the engine must own data loading; if the policy function makes ad-hoc queries, batching is impossible.
• Declared dependencies — the policy language must expose its required data via static analysis (Figma: ExpressionDef walk; Cedar: schema). Lambda-style arbitrary functions defeat this.
• Three-valued evaluation — the expression evaluator must return null/unknown when data is missing, not crash or silently return false.

Design knobs

• Partition strategy. Load-plan heuristics are the most impactful knob. Common strategies:
• Coverage-ranked — most-commonly-determining data first (Figma's approach).
• Cost-ranked — cheapest queries first; medium-coverage shallow queries may beat high-coverage expensive ones.
• Fan-out-by-policy — every policy's entire dependency set first, then add policies in order of historical hit rate.
• Batch granularity. Smaller batches = more opportunities to short-circuit, but higher per-batch evaluator cost. Figma partitions by table-dependency groups.
• Parallelism within a batch. Load the batch's multiple queries in parallel to minimize per-batch latency.

Caveats

• Requires accurate three-valued evaluation. A bug that returns false when data is missing instead of null produces silent over-denial. Mitigation: evaluator test suite with explicit pending-state fixtures.
• Not all policy shapes short-circuit well. An expression that's a top-level or of N sub-expressions needs all sub-expressions determined as false to return false — worst-case still loads everything. Deny-overrides-allow helps here because any one deny going true ends the evaluation.
• Engine complexity cost. Partitioning, batch loading, iterative evaluation is more code than "load everything, then evaluate once." Pay it only when the hot path matters.

Seen in

• sources/2026-04-21-figma-how-we-built-a-custom-permissions-dsl — canonical instance. Explicit partitioned-load-plan description, worked example of ["team.permission", "=", "open"] being false making an enclosing and false without loading file / project, "more than halved" speedup.
• sources/2026-04-03-github-the-uphill-climb-of-making-diff-lines-performant — web-UI specialization, same conceptual pattern. GitHub's PR Files-changed tab streams diff content + background-fetches remaining files so users can interact with what's already arrived rather than waiting for a massive diff to finish loading. "Users are now able to see and interact with content sooner. No more waiting for a massive number of diffs to finish loading." Partner pattern to patterns/server-hydrate-visible-only on the SSR side, and to client-side virtualization on the rendering side. Shares the short- circuit framing of the Figma permissions-DSL case — make the common case (the first visible window) fast and let the rest stream behind it — at a different substrate (web UI rendering vs authorization-expression evaluation).