Skip to content

PATTERN Cited by 1 source

External secrets-manager interpolation

Pattern

At pipeline invocation time, the runtime looks up named secrets from an external key-management backend (AWS Secrets Manager, Azure Key Vault, GCP Secret Manager, Redis, etc.) and interpolates them into the pipeline configuration wherever a secret-reference placeholder appears. The pipeline's YAML / configuration file stays free of credential values — only references to secret names.

Canonical instantiation: Redpanda Connect's rpk connect --secrets flag (2024-12-03, Redpanda 24.3 release):

"Redpanda Connect now allows the rpk connect CLI flag --secrets to interpolate secrets values from external systems like AWS Secrets Manager, Azure Key Vault, GCP Secret Manager, and Redis. No more fumbling with different sets of environment variables or embedding credentials in your YAML!"

(Source: sources/2024-12-03-redpanda-redpanda-243-extends-lakehouses-with-streaming-data-cdc)

Why the pattern exists — what it replaces

Pre-pattern credential-handling shapes, all of which the pattern displaces:

  • Credentials embedded in YAML committed to source control. Worst case. Full credential exposure on repo breach, on backup leak, on CI-artefact exposure.
  • Credentials injected via environment variables per deployment. Better than YAML-embedded but creates a "different sets of environment variables" operational burden across environments, and env vars are readable by any process running under the same user.
  • Credentials fetched at runtime by application code. Works but requires each component author to re-implement the fetch + refresh + error-handling discipline.

The external-secrets-manager-interpolation pattern centralises the fetch+refresh+audit concern in the runtime, and constrains the credential-value flow to secrets-manager → runtime memory → pipeline. The configuration file never contains the value.

Sibling patterns in the credential-handling space

  • concepts/file-based-credential-store — alternative shape for substrate-level auth primitives (Oracle Wallet, Java keystore) where the credential is a file on disk, not a named secret in an external service. Canonical instance in Redpanda Connect Oracle CDC.
  • concepts/short-lived-credential-auth — orthogonal dial on credential lifetime; can compose with this pattern (secrets-manager minting short-lived tokens that the runtime interpolates).
  • concepts/token-vault — canonicalised in the 2026-04-14 Openclaw is not for enterprise scale post as the agent-credential-broker altitude; external-secrets-manager interpolation is the connector/pipeline altitude analogue for data-integration workloads.

Operational properties

  • Runtime-agnostic to backend. Same --secrets invocation surface regardless of which backend the operator chose — the runtime handles the backend-specific authN + fetch.
  • No credential-value persistence. Values live only in process memory; never written to disk by the runtime, never logged.
  • Backend failure = pipeline startup failure. If the secrets-manager is unavailable at rpk connect invocation, the pipeline can't start. This is load-bearing on the availability calculus: secrets-manager becomes a hard critical-path dependency for every Redpanda Connect deployment that uses the flag.
  • Backend auth is out-of-band. The runtime itself needs credentials to talk to the secrets manager — typically cloud IAM role assumption (AWS Secrets Manager via EC2 instance profile / IRSA), Managed Identity (Azure), Workload Identity (GCP). Redis requires either network-level isolation or its own auth layer.

Fully-managed variant (Redpanda Cloud)

The origin post previews a fully-managed console-integrated variant for Redpanda Cloud:

"Look for a fully managed version of secrets management (coming soon) built into the console for all Redpanda Cloud deployments."

Not shipping at post time. No post-launch wiki source has re-visited the console flavour.

Seen in

Last updated · 550 distilled / 1,221 read