PATTERN Cited by 1 source

Pool rebuild on credential rotation

The pattern

When a connection pool authenticates via short-lived credentials (OAuth tokens, temporary passwords, STS session tokens), pooled connections become invalid on credential expiry. The pool cannot simply refresh individual connections — it must detect credential change and rebuild the entire pool while draining in-flight queries on the old one.

Mechanism (as seen in Databricks retail-app)

_ensure_pool():
  if pool exists AND token unchanged → return pool (fast path)
  acquire lock (double-check locking)
    if token changed:
      schedule old pool close after 30s grace period
      build new pool with fresh token
      swap reference
  release lock

Key properties:

1. Fast-path dominance. Most calls hit the "pool exists, token unchanged" branch — no lock, no allocation.
2. Double-check locking. A mutex prevents N threads from all rebuilding the pool simultaneously on a token rotation event.
3. Grace period for in-flight queries. The old pool isn't killed immediately — a 30-second timer lets in-progress queries complete before connections are severed.
4. Token-as-password model. The OAuth access token is used as the Postgres password, and the client ID as the username. When the token rotates, all existing connections carry a now-invalid password.

When to use

• Database auth via OAuth / short-lived tokens (e.g., Lakebase, cloud-managed Postgres with IAM auth).
• Any pooled resource whose credential is time-bounded — HTTP connection pools to token-authenticated APIs, gRPC channels with rotating mTLS certs.
• Model-serving containers that hold long-lived pools but authenticate via platform-managed tokens.

Trade-offs

Advantage	Cost
No long-lived database passwords	Pool rebuild latency on rotation (one-time, amortised)
Automatic rotation without app restart	Complexity of double-check locking + grace drain
Works with standard connection pool libraries	Must wrap pool creation in custom lifecycle manager

Seen in

• sources/2026-07-16-databricks-what-happens-in-the-milliseconds-after-you-tap-pay — Both the FastAPI backend and the MLflow model container maintain ThreadedConnectionPool instances to Lakebase with this pattern: _ensure_pool checks token freshness, rebuilds with lock-guarded swap, and drains old pool on a 30-second timer.