PATTERN Cited by 1 source
Composable transformation pipeline¶
Intent¶
Decompose complex file/data processing tasks into small, reusable transformation steps that can be composed into pipelines for different use cases — instead of building monolithic per-format/per-product processing systems.
Problem¶
When a platform supports many input formats (hundreds) and many output types (thumbnails, previews, text, metadata, streaming manifests), building a separate end-to-end processor per format × output combination becomes unmanageable. Duplicate logic proliferates, configurations drift, package versions skew, and operational burden grows linearly with the product of formats and outputs.
Solution¶
Break each processing task into a sequence of primitive transformations that can be shared:
PowerPoint → PDF → page-images
Word → PDF → page-images (reuses PDF→image step)
PDF → page-images (reuses same step directly)
Video → transcode → streaming-manifest
Document → text-extraction (shared by search + AI)
Each primitive is: - Independently deployable and scalable - Owned by one team (or one plugin) - Cacheable at input/output boundaries - Composable into arbitrary pipelines by a coordinator
Consequences¶
Benefits: - New file formats or output types require adding one primitive, not rebuilding a pipeline. - Improvements to a shared step (e.g., faster PDF rendering) benefit all consumers simultaneously. - Caching at step boundaries eliminates redundant computation across products. - Each step has a single scaling profile and failure domain.
Costs: - Coordination layer adds latency and complexity. - Step boundaries must be carefully chosen — too fine-grained increases serialization overhead; too coarse reduces reuse. - Intermediate formats (e.g., PDF as interchange) become implicit contracts that are hard to change later.
Known instances¶
- systems/dropbox-riviera — >100 capabilities, >300 file formats, hundreds of thousands of transformations/second. The canonical large-scale instance documented in the wiki (Source: sources/2026-07-20-dropbox-riviera-universal-content-processing-platform).