CONCEPT Cited by 1 source

Runtime API vs GitOps source of truth

Definition

The runtime-API-vs-GitOps source-of-truth tension arises when a platform exposes both a declarative Git-reconciled substrate (e.g. Argo CD syncing manifests) and a runtime CRUD API that mutates live state (e.g. a REST API to add / remove pipelines). If API callers derive their intent from Git (a CI pipeline, a GitOps controller watching a config repo), the API is compatible with GitOps. If humans or external systems mutate state through the API without updating Git, the API becomes an anti-pattern — "snowflake" state accumulates that cannot be reconstructed from version control, breaking the fundamental GitOps guarantee that Git is the single source of truth.

Canonical framing

From the 2025-12-02 Redpanda GitOps tutorial describing the Redpanda Connect Streams-mode REST API:

"This API is compatible with the GitOps philosophy as long as it's used by automation that derives its desired state from Git. For example, a GitOps controller or CI/CD workflow can watch a repository of pipeline configs and apply them to Redpanda Connect through the API, just as Argo CD interacts with the Kubernetes API."

"It becomes a GitOps anti-pattern only when humans or external systems modify pipelines through the API without updating Git. This creates 'snowflake' pipelines that cannot be reconstructed from version control and breaks the fundamental GitOps guarantee: Git is the single source of truth."

(Source: sources/2025-12-02-redpanda-operationalize-redpanda-connect-with-gitops)

Why runtime APIs exist alongside GitOps

The runtime API is often the mechanism by which a GitOps controller applies desired state. Argo CD reconciles Kubernetes manifests by calling the Kubernetes API (a runtime CRUD API); the Kubernetes API is GitOps-compatible because Argo CD (not a human) is the usual caller. The same pattern applies to application-level APIs — if a GitOps-style controller watches a pipeline-config repo and POSTs to the Redpanda Connect Streams API whenever config changes, the API becomes a reconciliation mechanism rather than a drift-inducing surface.

Runtime APIs also provide:

• Low-latency introspection — GET /streams, GET /metrics for dashboards and health checks
• Debugging flexibility — operators can inspect live state without going through the Git loop
• Ecosystem integration — CLI tools, IDEs, and programmatic clients can interact with the runtime without a full repo context

The question is not "do we expose a runtime API?" but "is every mutation through the API derivable from Git?"

Failure mode — snowflake pipelines

Without discipline, the typical failure sequence is:

1. Pipeline is created via Git commit → CI / CD / GitOps controller → runtime API → live state matches Git
2. An operator needs to "quickly debug something" — hits the runtime API directly with curl to tweak a pipeline parameter
3. Live state now diverges from Git; Git-derived reconciliation either reverts the change (if the controller has selfHeal: true) or leaves the drift in place (if drift detection is manual)
4. If drift is reverted, the operator's debug change disappears mid-session, frustrating debug
5. If drift is left, the pipeline becomes a snowflake — no one remembers the live tweak, it's not in version control, and any future Git-driven rollout risks overwriting it or preserving it inconsistently

Generalised form: two writers to one substrate, only one of which is Git-backed, leads to irreconcilable state unless the non-Git writer is disabled or the state is one-way sync-merged.

Mitigation shapes

Option A — derive all API calls from Git. CI / CD / GitOps controller is the only caller. Humans edit Git; their intent flows through automation. Preserved GitOps discipline at the cost of velocity on ad-hoc changes.

Option B — make the API read-only for humans. API auth gates writes behind service-account credentials that only automation holds; humans can GET but not POST. Preserves discipline, loses debug flexibility.

Option C — runtime write-back to Git on API mutation. Every API-driven mutation produces a Git commit describing the change (perhaps on a separate runtime-mutations branch). Merged manually. Preserves Git as source of truth at the cost of repo churn.

Option D — accept drift, with strong detection. Let humans mutate via API; run a continuous drift detector that surfaces divergence as alerts. Operators resolve drift by either promoting the runtime mutation to Git or reverting. Lower-discipline operational mode, but practical for debug-heavy environments.

Option E — disable the runtime API in production. Only Standalone mode in prod, no Streams-mode API exposure. Simple, pessimistic.

Seen in

• sources/2025-12-02-redpanda-operationalize-redpanda-connect-with-gitops — canonical wiki introduction. Redpanda explicitly frames the Streams-mode REST API as GitOps-compatible "as long as it's used by automation that derives its desired state from Git" and anti-pattern "only when humans or external systems modify pipelines through the API without updating Git." The tutorial doesn't ship an automated guardrail (no admission webhook, no API auth gate) — the anti-pattern framing is discipline, not an invariant.

Related

• concepts/gitops — the substrate whose single-source-of-truth property is at risk
• concepts/standalone-vs-streams-mode — Standalone mode has no runtime API and is therefore GitOps-aligned by construction; Streams mode introduces the tension
• systems/redpanda-connect — canonical runtime where the Streams REST API coexists with GitOps-driven pipeline management
• systems/argocd — the GitOps controller whose reconciliation loop the API can either complement (as mechanism) or undermine (as drift source)