Skip to content

PATTERN Cited by 1 source

gRPC-over-Unix-socket language-agnostic plugin

Pattern

Extend a host application with plugins written in any language by running each plugin as a separate OS subprocess, communicating with the host over a [[systems/ grpc|gRPC]] service exposed on a Unix domain socket. The gRPC service interface mirrors the host's internal component / operator / middleware abstraction, so a plugin speaks the same protocol whether implemented in Go, Python, Rust, Java, or anything else with a gRPC code generator.

Canonical wiki instance

Redpanda Connect dynamic plugins (2025-06-17 Beta, v4.56.0, Apache 2.0). Redpanda Connect's previous plugin model required plugins to be written in Go and compiled into the main binary; the new shape ships external plugins as subprocesses of arbitrary language that the host talks to over gRPC on a Unix socket. Three built- in wrapper plugins (BatchInput / BatchProcessor / BatchOutput) act as dispatch shims in the host, and the cross-process gRPC service "closely mirrors the existing interfaces defined for plugins within Redpanda Connect's core engine, Benthos" (Source: sources/2025-06-17-redpanda-introducing-multi-language-dynamic-plugins-for-redpanda-connect).

Structural elements

Every instance of the pattern has these parts:

  1. Plugin descriptor — YAML / JSON / CLI-flags file that declares the plugin's name, type (which abstract interface it implements), and the command argv array the host uses to launch the plugin subprocess. Redpanda Connect's plugin.yaml is the canonical shape.
  2. Host-side dispatch shim — a first-class component in the host process (implemented in the host language) that owns the subprocess lifecycle and translates internal host calls into gRPC calls to the plugin. For Redpanda Connect this is the three compiled Batch* wrapper plugins.
  3. gRPC service definition — .proto schema mirroring the host's internal plugin-interface abstraction (Benthos interfaces, for Redpanda Connect). One service per plugin type.
  4. Unix domain socket as the transport — per plugin subprocess, one socket. Not TCP / not HTTP/2 over TCP — Unix socket (cheaper, no port management, filesystem-level permissions).
  5. Language SDKs — at least one idiomatic SDK per supported language wrapping the gRPC server boilerplate. Redpanda Connect ships Go + Python SDKs at launch; the Python SDK's @redpanda_connect.processor decorator is the canonical ergonomic shape.
  • HashiCorp go-plugin (the lineage this pattern most directly descends from) — HashiCorp has used this model since ~2016 for Terraform provider plugins, Vault secret backends, Nomad task drivers. Same shape: subprocess + gRPC + handshake protocol. Redpanda Connect's dynamic plugins framework uses the go-plugin lineage ("the proven Go-plugin framework" in the post's Conclusion).
  • Envoy external processor (ext_proc) filter — an Envoy proxy filter that streams requests to an out-of-process gRPC service for modification. Same subprocess + gRPC pattern applied to HTTP request-processing.
  • Kubernetes CRI / CNI / CSI — container-runtime-interface / networking / storage pluggability in Kubernetes is subprocess + gRPC over Unix socket. The kubelet talks to containerd over /run/containerd/containerd.sock with gRPC; same shape, different domain.
  • Language Server Protocol (LSP) — editors speak JSON-RPC over stdio to an LSP subprocess. Structurally similar (subprocess + protocol boundary) but uses JSON-RPC over stdio rather than gRPC over Unix socket. Same language-agnosticism property.

Why Unix socket (not TCP, not stdio)

  • Filesystem-level permissions. Unix sockets respect Unix permissions / ACLs. Only processes with the right UID can connect. TCP on localhost has no such mechanism without additional auth.
  • No port management. No range to reserve, no conflict with other services, no firewall rules. A path like /tmp/plugin-<pid>-<name>.sock is unique per invocation.
  • Lower overhead than TCP. Unix socket bypasses the network stack — no IP layer, no port translation, no netfilter. Faster than loopback TCP by a measurable margin.
  • Richer than stdio. Stdio is inherently a single byte stream; multiplexing N concurrent gRPC calls requires a framing layer. Unix sockets are first-class bidirectional, and gRPC handles the framing natively.

Why gRPC (not custom JSON-RPC)

  • Protobuf serialization is faster and more compact than JSON. At high message rates across the boundary, this matters.
  • Generated SDKs for many languages. The whole pattern's language-agnosticism depends on the RPC framework having broad language support. gRPC has official or community clients for Go, Python, Rust, Java, C++, TypeScript, Ruby, Kotlin, Swift, etc.
  • Streaming support. gRPC supports client-streaming, server-streaming, and bidirectional streams natively — useful when the plugin is a source (server-stream) or a sink (client-stream).
  • Typed interface from a .proto. Schema evolution rules are well-defined; the plugin protocol is self-documenting.

When to apply

  • Host application that wants to be extensible at runtime without requiring extenders to use the host's language.
  • Plugin failures must not bring down the host (subprocess isolation is the containment property).
  • Plugin authors will come from a variety of backgrounds (data-science Python, ML-ops Python, Go, Rust) and the host wants to reach them all.
  • Plugins are per-deployment / per-pipeline configuration, not hot-path request handlers.

When not to apply

  • Hot-path per-request extension where the IPC cost is unacceptable. Use compiled in-process plugins instead — compiled-vs- dynamic plugin tradeoff documents the rule.
  • Plugin needs tight shared-memory access to host data. The boundary gets in the way. Consider an in-process embedding or FFI instead.
  • No batching upstream. If records arrive one-at-a-time with a strict latency budget, amortization fails and IPC dominates. See [[concepts/batch-only-component-for-ipc- amortization]].

Implementation tips from the canonical instance

  • Mirror the host's internal plugin interface in the gRPC .proto — don't invent a new abstraction. Redpanda Connect's gRPC service mirrors Benthos's existing plugin interfaces; that's the same abstraction developers already learn from the host.
  • Expose only batch variants if per-call IPC cost matters. Redpanda Connect exposes BatchInput / BatchProcessor / BatchOutput only; see [[concepts/batch-only-component- for-ipc-amortization]].
  • Ship an SDK per supported language, not just a .proto. Users should not have to write their own gRPC server scaffolding. The @redpanda_connect.processor decorator + processor_main() helper is the ergonomic Python shape.
  • Keep the compiled-plugin path available as the performance escape hatch. Redpanda Connect's guidance is explicit — dynamic plugins are flexible, compiled plugins are fast, use both.

Seen in

Last updated · 470 distilled / 1,213 read