Skip to content

REDPANDA 2025-12-02 Tier 3

Read original ↗

Redpanda — Operationalize Redpanda Connect with GitOps

Summary

Redpanda's 2025-12-02 unsigned tutorial post walks through the canonical GitOps deployment shape for Redpanda Connect pipelines on Kubernetes — using Argo CD as the continuously-reconciling GitOps controller, Helm for the Redpanda-authored chart, and Kustomize to wrap Helm + generate hashed ConfigMaps that trigger rolling pod restarts when pipeline configs change. The post is tutorial-voice pedagogy rather than a production retrospective — no fleet sizes, no latency distributions, no customer case studies — but canonicalises four design primitives the wiki's prior Argo CD + GitOps coverage had only stub-level anchoring for: (1) Standalone vs Streams deployment modes for Redpanda Connect with a decision matrix on isolation vs multi-tenant flexibility; (2) an Argo CD multi-source Application pattern ( split Helm chart repo + values repo) decoupling chart maintenance from config ownership; (3) Kustomize wraps the Helm chart (patterns/kustomize-wraps-helm-chart) using configMapGenerator + helmCharts + --enable-helm to produce a single reconcilable manifest; (4) Streams-mode REST API ≠ GitOps anti-pattern (concepts/runtime-api-vs-gitops-source-of-truth) — the API is GitOps-compatible only when automation derives its desired state from Git; becomes an anti-pattern when humans mutate pipelines through the API without updating Git. Closing product roadmap lists maturation ideas: automatic linting/testing, policy + compliance checks, developer-portal integration, external secrets, reusable pipeline template catalog, resource usage limits, multi-cluster scaling. Observability layer (kube-prometheus-stack + Prometheus service monitor + Redpanda Connect Grafana dashboard) deployed via a parallel Argo CD Application.

Key takeaways

  1. Standalone vs Streams mode — two deployment shapes for Redpanda Connect, with a verbatim comparison table canonicalising the trade-off axes: deployment model (pipeline config baked into values.yaml vs pipelines loaded from ConfigMaps at runtime); best fit (predictable isolated workloads vs multi-tenant/dynamic/ high-churn pipelines); scaling axis (replica count vs pods OR pipelines); config changes (pod rollout required vs triggered via ConfigMap hash); operational overhead (low + simple vs very low for large fleets + extremely flexible); Runtime API (none vs Streams REST API); GitOps alignment (always vs only when API derives from Git). TL;DR verbatim: "Choose standalone mode for simplicity and isolation. Choose streams mode for flexibility and scale, especially in platform-style deployments." (Source: sources/2025-12-02-redpanda-operationalize-redpanda-connect-with-gitops)

  2. Argo CD multi-source Application with Helm chart + values repo split — canonical multi-source pattern for decoupling chart maintenance from app config. Verbatim Application spec ships two sources: (1) Helm chart at repoURL: https://charts.redpanda.com, chart: connect, targetRevision: 3.1.0; (2) project repo (customer's fork) with ref: values. The Helm valueFiles entry references the values source via $values/standalone/standalone-mode.yaml. Rationale verbatim: "This pattern decouples maintenance of the Helm chart from your application configuration." Sync policy is automated: { prune: true, selfHeal: true }. (Source: sources/2025-12-02-redpanda-operationalize-redpanda-connect-with-gitops)

  3. Kustomize wraps the Helm chart to produce ConfigMaps + rolling restart on hash change — architectural composition canonicalising Kustomize as the layer that Argo CD auto-selects when kustomization.yaml is present in the source directory. The verbatim kustomization.yaml uses configMapGenerator to turn each pipeline file (config/first-names.yaml, config/last-names.yaml) into a ConfigMap with a hash suffix (disableNameSuffixHash: false) and helmCharts to inflate the Redpanda Connect chart with valuesFile: streams-mode.yaml. Two preconditions: (a) enable Helm in Kustomize on Argo CD — kustomize.buildOptions: --enable-helm --load-restrictor LoadRestrictionsNone; (b) restart the repo-server + application-controller. Mechanism verbatim: "Kustomize generated a new hashed ConfigMap name when the pipeline configuration changed and rewrote the Helm chart references so the deployment points to the new hash. ArgoCD then executed a rolling restart with graceful component shutdown to minimize the data loss." (Source: sources/2025-12-02-redpanda-operationalize-redpanda-connect-with-gitops)

  4. Streams mode REST API is GitOps-compatible only when automation derives desired state from Git — canonical anti-pattern framing. Streams mode exposes an HTTP REST API to CRUD pipelines at runtime (/version, /ready, /streams, /metrics over kubectl port-forward svc/redpanda-connect-streams 8081:80). Verbatim condition: "This API is compatible with the GitOps philosophy as long as it's used by automation that derives its desired state from Git. For example, a GitOps controller or CI/CD workflow can watch a repository of pipeline configs and apply them to Redpanda Connect through the API, just as Argo CD interacts with the Kubernetes API." Verbatim anti-pattern: "It becomes a GitOps anti-pattern only when humans or external systems modify pipelines through the API without updating Git. This creates 'snowflake' pipelines that cannot be reconstructed from version control and breaks the fundamental GitOps guarantee: Git is the single source of truth." (Source: sources/2025-12-02-redpanda-operationalize-redpanda-connect-with-gitops)

  5. Observability stack as parallel GitOps-managed Application — canonical instance of deploying kube-prometheus-stack (Prometheus + Alertmanager + Grafana + out-of-the-box K8s dashboards) via a parallel Argo CD Application. Redpanda Connect natively exposes Prometheus-compatible metrics — "without custom exporters or sidecars". A Prometheus service monitor + the Redpanda Connect Grafana dashboard bootstrap real-time telemetry without manual configuration. Framing verbatim: "The entire observability layer is declarative, version-controlled, self-healing, and baked directly into your platform's lifecycle. No manual dashboard configuration or ad-hoc commands required." (Source: sources/2025-12-02-redpanda-operationalize-redpanda-connect-with-gitops)

  6. Scaling shape for standalone mode = replica-count Git commit — verbatim workflow: edit deployment.replicaCount from 1 to 3 in standalone-mode.yaml, git commit -am "Scaling Up", git push, wait for Argo CD sync (or trigger manual). Scaling is a Git-commit operation, not a kubectl scale operation. Canonical instance of GitOps replacing imperative kubectl invocations with declarative PR merges. (Source: sources/2025-12-02-redpanda-operationalize-redpanda-connect-with-gitops)

  7. Scaling shape for streams mode = add pipeline to ConfigMap or add pods — Streams mode has two orthogonal scaling axes: (a) add/remove pipelines by editing the config/*.yaml files that Kustomize turns into a ConfigMap (which triggers rolling restart via hash change); (b) add pods by bumping deployment.replicaCount in streams-mode.yaml. Streams mode consumes pipeline config from ConfigMaps at runtime — no pod rebuild for config-only changes, only ConfigMap-hash-driven rolling restart. (Source: sources/2025-12-02-redpanda-operationalize-redpanda-connect-with-gitops)

  8. Decommission path preserves the Argo CD Application — verbatim operational guidance: "If you want to keep the Argo CD applications, then simply scale the pipeline replicaCount to 0. Otherwise, you can delete the entire application in the UI or CLI." Two decommission shapes preserve GitOps discipline: scale-to-zero (keep the Application so future enable is also a Git commit) vs argocd app delete (remove the Application entirely). (Source: sources/2025-12-02-redpanda-operationalize-redpanda-connect-with-gitops)

Systems

Concepts

Patterns

Operational numbers (none)

No production numbers disclosed. Post is tutorial-voice pedagogy — no fleet sizes, no latency distributions, no customer case studies, no scale claims. Canonical targetRevision: 3.1.0 pinned version of the Redpanda Connect Helm chart is the only concrete version number.

Caveats

  • Tutorial voice — this is how-to walkthrough pedagogy, not an architecture retrospective or production incident narrative. Architecture density is moderate (~30-40% of body) concentrated in the standalone/streams comparison table, the Argo CD multi-source Application spec, the Kustomize + Helm composition with --enable-helm precondition, and the Streams-mode REST API anti-pattern framing.
  • Zero production context — no fleet sizes, no latency distributions, no benchmark numbers, no customer references. Every architectural claim lives at tutorial altitude rather than retrospective altitude.
  • Tutorial uses EKS via HashiCorp Terraform — not production-shape EKS hardening; no VPC / security-group / IAM depth.
  • Secrets management — redpanda-password is loaded via kubectl create secret generic rather than external secrets manager; external secrets is name-checked in the closing roadmap but not demonstrated.
  • Standalone-to-Streams migration path not covered — the tutorial treats the two modes as parallel deployments. Production teams migrating from one to the other don't have a canonical migration recipe in this post.
  • API-mutation anti-pattern framing is prescriptive not enforced — the post warns against humans mutating Streams-mode REST API without Git update but ships no automated guardrail (no admission webhook, no API auth-gate tied to Git commit hash). The anti-pattern is a discipline, not an invariant.
  • Observability stack is additional Argo CD Application, not integrated into Redpanda Connect lifecycle — operator teams manage two Applications (Redpanda Connect + observability) with independent sync cycles. Cross-Application dependency coordination (observability-before-pipeline) not canonicalised.
  • No mention of Redpanda Operator path — the tutorial deliberately uses Helm + ArgoCD, skipping the Redpanda Operator path that the 2025-05-06 Kubernetes guide canonicalised. Post names the operator's Topic + User CRDs as "GitOps-compatible alternatives" for topic / user provisioning but doesn't demonstrate end-to-end operator + ArgoCD composition.
  • Unsigned — no byline; Redpanda default attribution.
  • Product-roadmap laundry list at the end (automatic linting, policy + compliance, developer portal, external secrets, template catalog, resource limits, multi-cluster) is aspirational, not shipped — signals what Redpanda believes a mature Redpanda-Connect GitOps platform needs but doesn't yet ship.
  • URL authoritative: raw file's url: field is https://www.redpanda.com/blog/operationalize-redpanda-connect-gitops (URL slug matches filename slug sans hash — both are the full-title slug form).

Source

Last updated · 470 distilled / 1,213 read