Skip to content

CONCEPT Cited by 2 sources

Bloblang

Bloblang is Redpanda Connect's built-in declarative mapping, filtering, and transformation language — a small DSL for per-record reshape operations inside a Redpanda Connect pipeline's processors block and for interpolating dynamic values into config fields (e.g. routing tables, subject names, filenames). Originated in the Benthos open-source project before Redpanda acquired it in 2023.

Official documentation: Bloblang reference.

Two surfaces: processor + interpolation

Bloblang shows up in Redpanda Connect in two distinct roles:

1. Inline mapping processor

As a processor, Bloblang is the declarative per-field mapping layer between input and output. The canonical idiom (Source: sources/2026-03-05-redpanda-introducing-iceberg-output-for-redpanda-connect):

pipeline:
  processors:
    - mapping: |
        root = this
        root.ingested_at = now()

Use cases named in the 2026-03-05 Iceberg output launch post verbatim:

  • "Reshape, filter, and enrich messages inline"
  • "Flatten nested JSON into a columnar-friendly schema"
  • "Strip sensitive fields before they reach the lakehouse"
  • "Derive new columns from existing ones"

2. Config-field interpolation

Bloblang expressions embedded in ${!...} tags inside config string values allow dynamic routing and content-dependent config. The canonical example from the 2026-03-05 Iceberg output launch post is multi-table routing:

output:
  iceberg:
    namespace: raw.events
    table: 'events_${!this.event_type}'

This routes messages to different Iceberg tables based on the event_type field — one pipeline, N destination tables. The wiki canonicalises this as patterns/bloblang-interpolated-multi-table-routing.

Named complement: Starlark

Per the 2025-04-03 Gallego autonomy essay, Bloblang is paired with Starlark (a Python subset embedded via github.com/google/starlark-go) as a two-tier programming surface: Bloblang for the declarative default, Starlark as the escape hatch when "Bloblang isn't expressive enough". The combined shape is the mechanism behind dynamic content filtering in the MCP pipeline Gallego describes as the future of fine-grained ACL enforcement.

Why it's load-bearing on the wiki

  • Multi-table routing from a single pipeline definition is the architectural capability that separates the Iceberg output from legacy "configuration hell" connectors that require rigid per-table mappings (2026-03-05 launch post framing).
  • In-stream PII stripping / flattening / column derivation is the pre-lakehouse cleanup story the 2026-03-05 post positions against running sidecar Flink jobs or separate transformation services.
  • Config-field interpolation makes one YAML pipeline content-addressable across destinations, subjects, topics, and file paths — the general dynamic-routing substrate across many Redpanda Connect output connectors.

Seen in

Last updated · 470 distilled / 1,213 read