CONCEPT Cited by 1 source

Subprocess plugin isolation¶

Subprocess plugin isolation is the design choice to run a plugin as a separate OS process from the host application, with an IPC protocol (typically gRPC over a Unix domain socket, or similar) as the only coupling between them. The defining property: the plugin subprocess and the host share no address space, so a crash (segfault, unhandled exception, language- runtime failure) in the plugin cannot corrupt or terminate the host process.

Canonical wiki instance¶

Redpanda Connect dynamic plugins (2025-06-17) — "Plugins run in separate processes, so crashes won't take down the main Redpanda Connect engine." (Source: sources/2025-06-17-redpanda-introducing-multi-language-dynamic-plugins-for-redpanda-connect). Each plugin maps to one subprocess; the host (Go-compiled Redpanda Connect engine) communicates with it over gRPC on a Unix socket.

Why this matters — the fault-containment argument¶

Two classes of failure the subprocess boundary contains:

Memory-safety violations in the plugin. A plugin linking native C extensions (NumPy, PyTorch, Pillow) can segfault. If the plugin runs in-process with the host, the whole server dies. As a subprocess, only the plugin dies — the host observes EOF on the socket and can restart or fail the pipeline gracefully.
Language-runtime panics in the plugin. A Python plugin throwing an uncaught exception, or a JVM running out of heap, takes down the interpreter hosting it. With in-process embedding (ex: CPython embedded in a Go process via cgo) the failure mode is often a full-process abort. As a subprocess, the failure is local.

The contrast is with the in-process plugin model — the classic dynamic-library approach (Go's plugin stdlib, shared- object .so dlopen, JVM classloader plugin) — where the plugin runs in the host's address space. Failures there are not containable without process-level isolation.

Cost of the boundary¶

The subprocess boundary is not free:

IPC overhead per call. Every host→plugin call is a protobuf encode + Unix socket write + context switch + protobuf decode on the other side. Paid on every message in the worst case. Mitigated by batching so the IPC cost is amortized over many messages.
No shared memory. Zero-copy tricks (sharing a buffer across the boundary) require explicit shared-memory segments or file descriptors; don't happen by default.
Lifecycle management. The host must spawn, supervise, and reap plugin processes. A crashed plugin leaves a dead socket file; the host's supervisor must clean up and optionally restart.
Memory resident cost. Each plugin subprocess has its own interpreter / runtime footprint (tens to hundreds of MiB for a Python plugin with ML libraries). In-process plugins share the host's runtime.

The trade-off is architectural: you pay the subprocess cost on every batch to buy language agnosticism + crash containment. The compiled-vs- dynamic tradeoff captures when to pay it.

Contrast with in-process plugin failure modes¶

The Go stdlib plugin package is the archetype of the in-process alternative: shared address space, shared runtime, method calls are regular Go function calls at native speed. But importing plugin at all imposes linker- level costs (no dead-code elimination, forced method retention) — and a crash in the plugin terminates the host. The subprocess model escapes both the linker cost (plugin is not a Go plugin, not linked into the host binary) and the crash cost (separate address space).

Generalization — when the subprocess boundary is worth it¶

The subprocess boundary is the right call when at least one of these holds:

Untrusted or multi-tenant plugin code. A plugin downloaded from a marketplace, or contributed by external developers, whose crash / resource-exhaustion behavior you cannot reason about. Process isolation is the minimum required containment (see also capability-based sandbox).
Plugin in a different language / runtime. An in-process plugin requires language embedding (cgo, JNI, Python C API); a subprocess plugin only requires an IPC protocol. The gRPC plus language SDK shape (patterns/grpc-over-unix-socket-language-agnostic-plugin) removes the language-embedding problem entirely.
Crash recovery is a feature. If plugin failures should not bring down the host — streaming data pipelines, long-running services with many pipelines — the blast-radius constraint (blast radius) demands isolation.

Not the right call when¶

Per-call cost dominates. A plugin invoked at request rate with small payloads and strict p99 budgets (hot path of a proxy, inner loop of a serializer) is a bad fit. Compiled plugins with in-process calls are ~orders of magnitude faster per invocation.
Plugin needs access to host's internal state. Shared data structures, host-process locks, or in-memory caches of the host cannot be exposed across the subprocess boundary without building an explicit IPC protocol for them.

Seen in¶

sources/2025-06-17-redpanda-introducing-multi-language-dynamic-plugins-for-redpanda-connect — canonical wiki instance: one subprocess per plugin, gRPC-over-Unix-socket protocol, "crashes won't take down the main Redpanda Connect engine" as the explicit reliability claim.