Skip to content

CONCEPT Cited by 1 source

Derived-key aggregation

Derived-key aggregation is the logging discipline of counting cryptographic operations on a KDF-derived child key against the parent keyset's name rather than against the child key's own name.

Canonical wiki framing from Meta's 2024-12-02 cryptographic monitoring post. FBCrypto supports "derived crypto" where "'child' keysets [are] derived from 'parent' keysets by applying a key derivation function (KDF) to all the keys in the keyset with some salt", used by features "that need to generate millions of keys."

Problem

Naive logging emits one row per unique (child-key-name, method, algorithm) tuple per flush. A feature that mints millions of child keys from one parent inflates the aggregation-key cardinality — the buffered logger degenerates toward per-event logging exactly for the features with the highest event rate. Meta disclosed that initial implementation "created a unique row in the buffered logger for every derived keyset, which used a lot of space and put increased load on backend data stores."

Fix

Aggregate child-key events under the parent-keyset name. The aggregation tuple becomes (parent-key-name, method, algorithm) instead of (child-key-name, method, algorithm) — cardinality collapses from O(child keys) to O(parent keysets).

Correctness property: pessimistic upper bound

The parent-aggregated counter over-counts any specific child key's usage (other children of the same parent contribute to the same counter). This is safe for key-overuse detection: if the parent counter is below the data-volume budget, every child is definitely safe; if the parent counter exceeds the budget, at least one child might be overused and the operator investigates. The alarm is pessimistic — it never misses an overused child — but loses the ability to pinpoint which child.

In Meta's framing: "In the worst case, the aggregations would be a pessimistic counter for any given child key."

Trade-offs

  • Cardinality compression at the cost of per-child attribution. You cannot ask "how many operations did child-key ABC do?" from the aggregated dataset — only "how many operations did the parent keyset do in total?"
  • Alarm semantics shift from per-key to per-parent. Alerting must be thresholded on the parent's aggregated count rather than on any single child. If per-child attribution is needed for forensics, a separate finer-grained log (rare-event sampling, explicit trace) is required.
  • Feature coverage. Applies only to KDF-derived features where the child keys are structurally related — doesn't generalise to independent keys with unrelated lifecycles.

General pattern: aggregation-key discipline

Derived-key aggregation is one concrete instance of the general discipline of choosing the aggregation key so that cardinality stays bounded relative to event rate. The buffer-and-flush pattern only works when the aggregation key has low cardinality relative to the underlying event rate; when it doesn't, compression fails and costs explode. Meta's derived-key case is a canonical example of a cardinality hot-spot in an otherwise low-cardinality key space, and the standard fix shape: re-partition the key upward to a coarser granularity that preserves the alarm semantics.

Seen in

  • sources/2024-12-02-meta-built-large-scale-cryptographic-monitoring — canonical wiki disclosure of derived-key aggregation as the FBCrypto-specific cardinality fix for the KDF-heavy child-key explosion. "Thanks to this aggregation, we were able to cut down on the vast majority of our logging volume, compared to the space that would have been used with no aggregation."
Last updated · 319 distilled / 1,201 read