Skip to content

PATTERN Cited by 1 source

Pipeline with open-ended passes

Summary

Structure a long-running transformation — code migration, static-analysis rewrite, format conversion — as an ordered sequence of named phases where any phase may accumulate arbitrary custom steps over time. There is no single transform that hits the long tail; the pipeline is the long-tail solution. Each step closes one corner case the generic transform doesn't handle; the aggregate of hundreds of steps over years is what makes the migration complete.

Why this shape wins at scale

For any non-trivial transformation on a real codebase, three observations hold:

  1. No single tool handles the whole job. A generic converter like J2K produces non-building output "on the vast majority of files" at Meta's scale. The custom frameworks, the generated code, the edge-case syntax — each creates a class of failure the upstream tool doesn't know about.
  2. Failure modes are discovered incrementally. Every unit of progress on the long tail reveals new ones. An up-front complete specification of all needed transformations is impossible.
  3. Ordering matters. Some transformations depend on earlier transformations having run (e.g. nullability fixes before translation; translation before Android-specific idiom polishing).

The open-ended-pass pipeline turns these into an architectural advantage: new steps are additive to the appropriate named phase, do not touch unrelated phases, and compose under the explicit ordering discipline. It is the architecture that admits "I found a new corner case" as a routine diff, not an architectural disturbance.

Canonical wiki instance — Meta's Kotlinator

Kotlinator has six named phases and >200 custom steps across three of them. The phases (quoted from the 2024-12-18 Meta post):

  1. Deep build — resolve symbols.
  2. Preprocessing — ~50 steps for nullability, J2K workarounds, custom DI.
  3. Headless J2K — the core converter.
  4. Postprocessing — ~150 steps for Android, further nullability, idiomaticity.
  5. Linters with autofixes.
  6. Build-error-based fixes — see concepts/build-error-driven-fix-loop.

Meta's explicit framing:

"Due to the size of our codebase and the custom frameworks we use, the vast majority of conversion diffs produced by the vanilla J2K would not build. To address this problem, we added two custom phases to our conversion process, preprocessing and postprocessing. Both phases contain dozens of steps that take in the file being translated, analyze it (and sometimes its dependencies and dependents, too), and perform a Java-> Java or Kotlin->Kotlin transformation if needed."

Design discipline

Four practices keep the pattern sustainable as step count grows:

  • Each step is independently ownable. A single person can reason about a single step without understanding the rest of the pipeline.
  • Steps declare their inputs + outputs. Orderings become checkable, not tribal.
  • Steps bail explicitly when they lack information (e.g. unresolvable third-party symbols). See concepts/metaprogramming-on-broken-code.
  • Steps are reviewed individually. A PR adding a step is a small, reviewable unit; the pipeline never sees a big-bang rewrite.

Extension opportunities

  • Open-source extension points. Meta's collaboration with JetBrains to add J2K client hooks (patterns/upstream-collaboration-as-migration-unblock) is the same discipline applied one level up — the pipeline exposes open-ended-pass shape to its users, not just within its own organisation.
  • Compiler-error feedback. Phase 6 of Kotlinator is itself a micro-pipeline with open-ended passes — one per known error class.

Seen in

Last updated · 517 distilled / 1,221 read