Skip to content

CONCEPT Cited by 1 source

Standalone vs Streams mode

Definition

Redpanda Connect ships with two deployment modes optimised for different operational profiles:

  • Standalone mode — runs a single pipeline with configuration baked directly into the deployment (values.yaml). Simple, predictable, ideal when pipelines need strong isolation or distinct scaling behaviour. Scaling is deployment.replicaCount — one value per pipeline, per deployment.

  • Streams mode — hosts multiple pipelines within a single deployment. Pipelines are loaded dynamically from Kubernetes ConfigMaps at runtime; pipelines can be added, updated, or removed without redeploying pods. Ideal for multi-tenant platforms, internal developer portals, or any environment with high pipeline churn. Exposes a Streams REST API (HTTP) for runtime CRUD.

Canonical comparison table (verbatim from 2025-12-02 tutorial)

Axis Standalone mode Streams mode
Deployment model Pipeline config baked into values.yaml Pipelines loaded from ConfigMaps at runtime
Best for Predictable, isolated workloads Multi-tenant, dynamic, or high-churn pipelines
Scaling Scale the entire pipeline by adjusting the replicaCount Scale by adding pods OR by adding/removing pipelines
Config changes Requires pod rollout Triggers rollout via ConfigMap hash changes
Operational overhead Low (simple, isolated) Very low for large fleets, extremely flexible
Runtime API No Yes — Streams REST API
GitOps alignment Always GitOps-aligned GitOps-aligned unless the API mutates pipelines without Git
Ideal use cases Strong isolation, separate resource profiles Per-tenant pipelines, ephemeral pipelines, centralised platform

Canonical TL;DR verbatim: "Choose standalone mode for simplicity and isolation. Choose streams mode for flexibility and scale, especially in platform-style deployments." (Source: sources/2025-12-02-redpanda-operationalize-redpanda-connect-with-gitops)

Why the modes exist as separate products

The two modes reflect distinct operational use cases, not two flavours of the same one:

  • Strong isolation vs resource sharing. Standalone gives each pipeline its own pod / deployment, with independent scaling and resource limits. Streams pools pipelines into a shared pool of pods, amortising the JVM-like base cost.
  • Static config vs dynamic pipeline set. Standalone bakes config at chart-install time; adding a new pipeline means a new deployment. Streams loads config from ConfigMaps at runtime; adding a new pipeline means editing a ConfigMap.
  • Rollout semantics. Standalone requires a pod rollout for every config change (because the pipeline config is a chart value). Streams gets a rollout only when the ConfigMap hash changes — which, under Kustomize's hash-suffixed ConfigMap names, is automatic.

Streams REST API as the bidirectional surface

Streams mode exposes an HTTP REST API (default port 4195 in containers, 8081 via port-forward in the canonical tutorial) supporting:

  • GET /version — engine version
  • GET /ready — readiness probe
  • GET /streams — list all pipelines
  • POST /streams/{id} — create / replace a pipeline
  • DELETE /streams/{id} — remove a pipeline
  • GET /metrics — Prometheus-format metrics

This API is the mechanism that makes dynamic pipeline management possible — but also the mechanism that introduces the runtime-API vs GitOps source-of-truth anti-pattern when operators forget that Git-as-source-of-truth means API calls must derive from Git commits, not from ad-hoc console clicks.

Decision framework

Choose Standalone when:

  • You have a small number of stable, long-lived pipelines
  • Pipelines have very different resource profiles (CPU vs memory vs network)
  • You need strong isolation — a runaway pipeline shouldn't affect neighbours
  • You don't need runtime pipeline management

Choose Streams when:

  • You have many pipelines (dozens to hundreds)
  • Pipelines are short-lived or frequently modified (ephemeral, per-tenant, or CI-triggered)
  • You're building a developer-portal-style platform where users self-serve pipeline creation
  • You need a centralised place to manage + observe all pipelines

Seen in

Last updated · 470 distilled / 1,213 read