Skip to content

PATTERN Cited by 1 source

Shared execution engine with pluggable executors

Pattern

Build a single execution engine that handles all runtime concerns — IO substrate, concurrency, retries, backpressure, configuration parsing — and expose a pluggable executor interface where the per-event-type business logic lives. Run the same engine + the same executor plugins in multiple runtimes (streaming, batch, serving) so the "shared definition" compiles to literally the same execution logic across runtimes — only the scheduling shape differs (Source: sources/2026-05-21-pinterest-making-user-sequence-data-more-cost-efficient-faster-and-easier-to-use).

Shape

                 ┌────────────────────────────────────┐
                 │   Engine / framework layer         │
                 │   ─────────────────────────────    │
                 │   • source connectors              │
                 │   • sink writers                   │
                 │   • concurrency / parallelism      │
                 │   • retries / idempotency          │
                 │   • backpressure                   │
                 │   • config parser + validator      │
                 │   • observability                  │
                 └────────────┬───────────────────────┘
                              │ executor.process(event)
                              ▼
                 ┌────────────────────────────────────┐
                 │   Executor / plugin layer          │
                 │   ─────────────────────────────    │
                 │   per event type:                  │
                 │   • filter predicate               │
                 │   • featurisation                  │
                 │   • raw → normalised mapping       │
                 │   • enrichment dispatch            │
                 └────────────────────────────────────┘

The engine is instantiated in each runtime with that runtime's source / sink / scheduling shape:

Runtime Engine input Engine output
Streaming Kafka topic Time-versioned columnar store (incremental writes)
Batch Warehouse table / log archive Columnar store (re-written partitions)
Online serving API request API response (request-time enrichments)

The executors are identical bytecode / source across runtimes — what changes is the framework that wraps them.

Pinterest's contract (verbatim)

"Framework responsibilities include wiring data sources and sinks, handling concurrency, retries, and backpressure, and parsing and validating configuration. Executors own the business-specific filtering and featurization logic and the mapping from raw events to normalized user-event representations."

"In plain terms, the executor is the 'business logic module' for a particular event type or grouping, while the execution engine handles everything around it."

Why this pattern works

The pattern aligns ownership with cadence:

Layer Owner Cadence
Framework Platform team Quarterly / yearly
Executor ML / product team Per event-type / per signal
Configuration ML / product team Per launch

Without the layer split, every event-type change becomes a pipeline-stack change; every framework change risks breaking individual event-type semantics.

The pattern also makes streaming + batch parity structural rather than aspirational. Pinterest's outcome:

"The shared engine allowed us to reuse the same core enrichment logic in both streaming jobs that handle near-real-time events and batch jobs that process historical data. That minimized code duplication and reduced drift between batch and real-time behavior."

This is the structural fix to the historical Achilles heel of lambda architecture: maintaining two code paths.

Where it fits

  • ML / data platforms with many event types + enrichments + workloads.
  • Multi-runtime substrates (streaming + batch + serving) that need to compute the same logical transformation.
  • Cross-team data pipelines where platform / domain ownership separation is a coordination requirement.

Where it doesn't fit

  • Workloads where business logic is thin and runtime concerns dominate (a Kafka-to-S3 sink with light enrichment doesn't need an executor abstraction).
  • One-off ad-hoc pipelines where configuration overhead exceeds business-logic complexity.
  • Stacks where the runtime concerns themselves are wildly heterogeneous (e.g. CPU-bound streaming + GPU-batch + RDMA-serving) — the shared engine becomes a leaky abstraction.

Anti-patterns

  • Framework leaks — adding event-type-aware logic to the framework (e.g. "if event type is X, retry 3x; else 5x") breaks the pattern. Either add a per-executor retry-policy hook or push the special case into the executor.
  • Executor reaches into IO — an executor that opens its own connection to Kafka or writes directly to S3 breaks the multi-runtime guarantee. Force IO through the framework.
  • Per-runtime executors — having a "streaming executor" and a "batch executor" for the same event type defeats the entire pattern. The executor must be runtime-agnostic.

Sibling patterns

Caveats

  • Executor discipline is hard to enforce mechanically. Linters can catch obvious framework leaks but the real boundary is judgment about "what runtime concerns belong here?".
  • Plugin loading semantics differ across runtimes. JVM streaming jobs vs Python batch jobs vs C++ serving jobs all load plugins differently; a "single executor" may need multiple language bindings or shared definitions enforced at compile time of each.
  • Performance ceiling differs across runtimes. Streaming runtimes can't pause the engine to call out to slow enrichment services on the critical path; batch runtimes can. The framework needs to expose "this enrichment is slow / async / cacheable" hints without leaking executor semantics back into the framework.

Seen in

Last updated · 542 distilled / 1,571 read