Skip to content

PATTERN Cited by 2 sources

Bloblang-interpolated multi-table routing

A single streaming-sink pipeline definition routes records to N destination tables based on per-record content, by embedding a mapping-language expression into the connector's table / namespace / destination config fields. The destination is content-addressed — the record's own fields determine where it lands — rather than statically mapped via operator-authored per-table config.

Canonical instance

The iceberg output connector for Redpanda Connect (shipped v4.80.0, 2026-03-05) supports Bloblang interpolation in both table and namespace config fields. Verbatim from the launch post (Source: sources/2026-03-05-redpanda-introducing-iceberg-output-for-redpanda-connect):

output:
  iceberg:
    namespace: raw.events
    table: 'events_${!this.event_type}'

A record with event_type: "signup" lands in table events_signup; event_type: "purchase" lands in events_purchase. One pipeline, one YAML file, N Iceberg tables — created on demand as new event_type values appear.

The foil: "configuration hell"

The Redpanda launch post names the architectural alternative verbatim:

"Traditional Iceberg connectors often lead to 'configuration hell', where every new table requires rigid mapping and brittle, manual updates. Suffer no more with Redpanda Connect!"

In a static-mapping shape (typical of Kafka Connect-era Iceberg sinks), onboarding a new event type requires:

  1. Operator discovers the new type exists.
  2. Operator writes a new connector config entry.
  3. Operator deploys the config change.
  4. Connector restarts / picks up new config.
  5. Records of the new type accumulate in a DLQ or get dropped during the interim.

The Bloblang-interpolated shape collapses all five steps into "new records appear → new table exists".

Composition with registry-less schema evolution

When this pattern composes with registry-less schema evolution, the combined shape is zero-operator-touch table fan-out: a new event_type value creates a new table whose schema is itself inferred from the first records landing there. This is the shape the Redpanda launch post targets for rapidly-evolving multi- tenant or multi-event-type streams.

Implementation mechanics (general)

Any mapping DSL capable of extracting record fields and interpolating them into a string can implement this pattern. Bloblang's ${!expression} syntax is one shape; Kafka Streams routeByKey(value.getSchema().getName()) is a functionally-equivalent programmatic shape in a different DSL.

The key architectural properties are:

  • Config-level dynamism — the destination is expressed as a template, not a fixed string.
  • Per-record evaluation — the template is evaluated on each record, so destinations can vary arbitrarily within a stream.
  • Automatic destination materialisation — the sink creates tables / topics / files that don't yet exist (vs erroring on undefined destinations).

The third property is the one that most distinguishes this pattern from superficially similar static-routing configs — static routing is effectively a switch statement, this is a pure mapping.

Trade-offs

  • Blast radius on template bugs — a Bloblang expression that unexpectedly returns "" for a class of records could land them all in a single unintended table, or fail pipeline-wide. Good hygiene is a fallback table or a dead-letter route.
  • Cardinality control — untrusted field values in the template can create unbounded tables (one per unique input). Production deployments typically sanitise / whitelist the interpolated values.
  • Catalog load — N tables have N REST-catalog entries and N manifest chains; per-table overhead multiplies with event_type cardinality. At high cardinality, a single table with event_type as a partition column may be the better design.
  • Query ergonomics — consumers who want a single logical view across events_* tables must write a union or use a catalog feature (Unity Catalog "volumes" / Glue views) — one-table-per- type has a discoverability cost.

Seen in

  • Redpanda — Oracle CDC now available in Redpanda Connect (2026-04-09) — second canonical instance at a different pipeline position. Where the 2026-03-05 Iceberg-output post canonicalises this pattern at the sink-to-table-per-event-type position, the Oracle CDC post canonicalises it at the CDC-source-to-topic- per-source-table position. The connector attaches the source table name to each record as metadata (meta("table_name")); the output config uses Bloblang interpolation (topic: ${! meta("table_name").lowercase() }) to route each CDC event to a per-table Kafka topic. Verbatim: "The table_name metadata field flows through the pipeline, and Bloblang interpolation routes each event to its own topic automatically. One pipeline config, any number of tables." Sibling-altitude canonicalisation to the Iceberg instance — the pattern shows up at both CDC-source and lakehouse-sink ends of streaming pipelines, composing one config → many destinations across dynamic source catalogues.

  • Redpanda — Introducing Iceberg output for Redpanda Connect (2026-03-05) — canonical wiki instance at the Iceberg-sink position. 'events_${!this.event_type}' worked example. Explicit foil against "configuration hell" of static per-table mappings.

Last updated · 470 distilled / 1,213 read