PATTERN Cited by 1 source

End-to-end seam validation

Intent

Detect integration failures at component boundaries ("seams") that pass all component-level tests, by exercising the full production path with representative inputs.

Problem

Component-level confidence creates false assurance at the seams. Each part of an ML pipeline — language detection, preprocessing, model, serving — may validate individually, but their combined behavior surprises in production. Per Kästner et al. (2021), ML components resist compositional reasoning because they carry no formal specifications; interactions can only be observed empirically.

Mechanism

1. Select a small, representative input set:
2. Traffic-weighted sampling from highest-volume segments
3. Deliberate over-representation of the tail (locales, input modalities, historically incident-prone patterns)

4. Explicit seeding with regression cases from prior incidents

5. Run inputs through the entire production path (not mocked components)

6. Measure quality and tail latency on the combined configuration

7. Use the same evaluation framework (deterministic cache) that powers component evaluation — so measurement is trustworthy

Example seam failures caught

• Language detection misclassifying code-mixed input
• Preprocessing truncating a field the model relied on
• Latency spikes from cache-warmth interactions
• Format/encoding mismatches between components

Design constraints

• Small enough to run efficiently on every release candidate
• Broad enough that seam-level bugs surface before deployment
• Reuses deterministic evaluation infrastructure (Layer 2) — no separate measurement system

Relationship to CACE principle

Sculley et al. (2015) identifies that ML components are entangled in ways that make isolated validation insufficient. This pattern operationalizes that insight with an infrastructure-level response.

Seen in

• sources/2026-07-14-airbnb-llm-evaluation-infrastructure — Airbnb's Layer 4: end-to-end validation over the full LLM production path