Skip to content

PATTERN Cited by 1 source

Artifact rides model-deploy pipeline

Problem

When a serving tier needs per-model configuration (feature allowlists, per-model routing weights, per-model traffic policies, per-model resource limits), the naive design ships the config through a separate config system with its own:

  • Artefact format
  • Distribution mechanism
  • Canary / rollout workflow
  • Rollback semantics
  • On-call runbooks

The separate config system then has to stay in sync with the model-deploy pipeline. Drift between the two — "new model version rolled out before its config did" or "config updated to new schema but model still on old version" — produces silent mis-scoring, untrimmed payloads, or crashed workers.

Solution

Package the per-model config artefact into the same bundle as the model itself, and ship it through the same staged delivery pipeline (canary → automated canary analysis → production → rollback if needed). Don't build a second control plane.

Pinterest's articulation

From the 2026-05-01 Feature Trimmer post (Source: sources/2026-05-01-pinterest-optimizing-ml-workload-network-efficiency-part-i-feature-trimmer):

"Treat the model signature as the source of truth … publish signatures as lightweight artifacts that can be consumed by deployment pipelines … aggregate per-model signatures into a per-bundle artifact that is deployed to the root alongside existing root configs … use the same staged delivery semantics as model rollout (canary, automated canary analysis, prod, rollback), so trimmer config changes ride the same operational rails as everything else."

The pipeline:

1. Deploy root configs  →  Canary
2. Deploy model configs →  Canary
3. Automated Canary Analysis (ACA)
4. Deploy root configs  →  Production
5. Deploy model configs →  Production

Root configs always lead so that when a new leaf model version lands, a matching allowlist is already available on the root. Pinterest explicitly flags this invariant: "We deploy root configs before rolling out new leaf model versions because the feature trimmer keys feature allowlists by model name + version. If a versioned request arrives without a matching allowlist, we skip trimming to avoid stale configs, which can cause a temporary rollout gap."

Additional invariant for rolling deploys: the root-configs artefact is backwards-compatible, containing allowlists for both current and pending versions, so requests hitting leaves mid-deploy always find a match.

When it fits

  • Per-model configuration changes in lockstep with the model. Feature allowlists, per-model rate limits, per-model shadowing policies, per-model instance-class pins.
  • The model-deploy pipeline already has well-tested canary / ACA / rollback infrastructure. Reusing it is cheaper than rebuilding it for a second artefact class.
  • The operational team is the same for models and their configs. Different teams would motivate separate pipelines.
  • The artefact is small (kilobytes of JSON, not gigabytes of additional data).

When it doesn't fit

  • Config changes at a different cadence than model changes. If configs evolve continuously while models refresh weekly, coupling them gives you either too-frequent unnecessary model redeploys or too-slow config updates. Consider a separate fast-path config system (e.g., patterns/config-separated-from-code-via-pubsub) instead.
  • Config is cross-model, not per-model. A global feature flag that affects all models belongs in a global config system, not bundled per-model.
  • Configs have different blast radius than models. If config changes need finer-grained rollout (per-cell, per-region, per-cohort) than models do, a separate system with its own rollout semantics is warranted.
  • Different approval workflow. If config changes have different review / compliance requirements than model changes, separate pipelines express the separation cleanly.

Failure modes

  • Artefact staleness during incremental signature-publishing rollout. Pinterest's bundle-build step handles this gracefully: "If the signature is missing, the pipeline logs a warning and skips that version rather than failing the entire build." The skip-on-miss at the bundle-build tier + the skip-on-miss at the serving tier compound to a safe rollout path.
  • Root-configs-lead violated. If the pipeline is reconfigured to deploy model configs first or to deploy both in parallel, the backward-compatibility window collapses and versioned requests can hit un-allowlisted leaves. Pinterest addresses this with explicit staging + ACA between stages.
  • Bundled-gitops-dependency anti-pattern (concepts/bundled-gitops-dependency-anti-pattern) — if the model waits on the config during build, or vice versa, in a way that couples their failure modes, a transient issue with one blocks both. Pinterest's "log warning and skip" posture explicitly avoids this.

Sibling patterns

Pinterest picks "ride the pipeline" because feature allowlists are strictly per-model-version — their correct value is fully determined by the model, so they should live with the model. Netflix picks "config via pubsub" because Switchboard Rules evolve independently of the routing service's code — the model-config here is human-authored policy, not mechanical-derived metadata.

Seen in

Last updated · 445 distilled / 1,275 read