Skip to content

PATTERN Cited by 1 source

Per-domain adaptive config learning

Problem

You need per-domain configuration for a classifier, normaliser, or behavioural policy, and:

  • The head of the distribution is tractable via curated rules — well-known platforms have stable, documented conventions; hand-written static rules cover them.
  • The long tail is not — there are orders of magnitude more long-tail entries than head ones, their conventions vary wildly, and hand-writing rules for each is infeasible.

Pinterest's framing (Source: sources/2026-04-20-pinterest-smarter-url-normalization-at-scale-how-miqps-powers-content-deduplication):

"For well-known e-commerce platforms, this can be solved with curated rules. Shopify URLs, for example, use variants as the key product differentiator. Salesforce Commerce Cloud uses parameters like start, sz, prefn1, and prefv1. For these platforms, static allowlists are sufficient. But Pinterest ingests content from a large number of domains, operating on a wide variety of platforms. For this long tail of domains, URL parameter conventions vary wildly. Static rules cannot scale to cover them all. We need a dynamic, data-driven approach."

Solution

Hybrid — head-curated + long-tail-learned:

  • Head: hand-written static rules (allowlists, regex, domain- specific configs) for well-known platforms where the human labour of authoring + maintaining rules pays off because each rule covers many domains.
  • Tail: adaptive learning — a system that derives per-domain configuration empirically from observed data, without per-domain human authoring.

Critically, the two operate in parallel, not as fallback. Both can contribute signal, and the combination (patterns/multi-layer-normalization-strategy) is usually more conservative than either alone.

Canonical instance — Pinterest MIQPS

MIQPS is the canonical production instance applied to URL query parameter classification:

Structural properties

The pattern has three load-bearing properties:

  1. Clear head/tail boundary — what goes into static rules is well-known platforms; what goes into learned config is everything else. No messy overlap territory where both fight.
  2. Adaptive tail scales with observation, not authoring — adding a new long-tail domain just means feeding its URLs into the pipeline; the config for it is learned automatically.
  3. Combined via OR semantics — see patterns/multi-layer-normalization-strategy. The ensemble is only as aggressive as the least aggressive layer.

When to apply

  • You have a per-entity configuration decision (per domain, per tenant, per site, per customer).
  • The distribution is heavily skewed: a small number of "head" entities account for a large fraction of traffic, while long-tail entities individually account for tiny fractions.
  • Head entities have documented, stable conventions that make static rules feasible.
  • Long-tail entities have enough observed behaviour to learn from, even if individually small.
  • The learning cost scales with entity count (not per-entity-request count), and the system can tolerate the resulting staleness (see concepts/offline-compute-online-lookup).

When not to apply

  • Per-tenant decisions with sensitive / mission-critical effect — if learning is wrong, failure mode is catastrophic, and you can't tolerate empirical error. Better: hand-authored rules across all tenants, possibly scoped.
  • Too little observation per entity — if most long-tail entities have only a handful of requests, there's nothing to learn from; the classifier defaults dominate the entries and the system is effectively static anyway.
  • Fast-moving conventions — if long-tail entity behaviour changes within an offline-cycle cadence, learned config is stale by publication time.

Generalisation

The pattern generalises to any long-tail configuration problem:

  • URL normalisation — Pinterest MIQPS (this post).
  • Per-merchant fee / pricing rules — curated for top merchants, learned for long tail.
  • Per-country checkout flow validation — hand-written for primary markets, inferred for new markets.
  • Per-device rendering quirks — curated for high-share devices, runtime-adaptive for long-tail devices.

Caveats

  • Two systems to maintain — static rules + learned pipeline. The head rules still need human review, and the learned pipeline needs its own observability + anomaly gating.
  • Boundary drift — as head entities grow or shrink, their classification (curated vs learned) may need to change. No automatic promotion/demotion mechanism in Pinterest's MIQPS post.
  • Learned-pipeline failure modes — the learned path inherits whatever failure modes the observation pipeline + learning algorithm have. Pinterest addresses these with patterns/conservative-anomaly-gated-config-update.

Seen in

Last updated · 319 distilled / 1,201 read