DATABRICKS 2026-07-22

Simplify AI agent orchestration with Lakebase Postgres¶

Summary¶

CLA (CliftonLarsonAllen LLP) and Databricks Forward Deployed Engineering built a production document-processing agentic system where Lakebase Postgres serves as the sole orchestration backbone — replacing the need for external message brokers (Kafka, Redis), schedulers (Airflow, Temporal), or caching layers. The system uses four Postgres-native patterns to transform a pair of relational tables into a crash-resilient, priority-aware, rate-limit-aware, idempotent task queue for long-running AI agent workloads. Reduces extraction time from hours to minutes.

Key Takeaways¶

Postgres-as-queue eliminates external infrastructure. Two tables (tasks + task_attempts) replace Kafka, Redis, Airflow, and Temporal. The parent-child relationship supports retries with per-attempt cost metadata, and the whole system is a single Postgres database that commits atomically with application state.
FOR UPDATE SKIP LOCKED enables concurrent priority-aware dequeue. Each worker locks the row it selects while other workers skip it. ORDER BY priority DESC, created_at preserves FIFO within priority levels. No external coordination service needed.
Lease-based crash recovery replaces heartbeats. Workers record a lease_expires_at timestamp at dequeue time. A periodic sweeper re-enqueues any task whose lease has passed — recovering terminated workers (VM eviction, OOM, deployment) automatically within minutes without coordination.
Three-mode rate-limit-aware throttling at dequeue time. (a) Concurrency cap: count PROCESSING rows, block if at MAX_CONCURRENT_TASKS. (b) Token budget: sum projected token rate across in-flight tasks, block if adding new task exceeds MAX_TPM. (c) Combined: apply whichever constraint is tighter. All decisions happen inside the same transaction as the dequeue lock.
Idempotent webhook callbacks prevent double-processing. The callback handler accepts both PROCESSING and ENQUEUED states, treats already-terminal tasks as no-ops. Identical payloads produce identical results — eliminating double-billing risk from network retries.
LISTEN/NOTIFY + SSE for real-time dashboard. State changes fire Postgres LISTEN/NOTIFY; backend fans out over Server-Sent Events. ~1s latency from state change to UI update. Permanent polling fallback at 10s interval handles silently-dropped SSE connections.
Per-application cost attribution via job-run-ID filtering. The orchestrator tracks which Databricks Job run IDs it submitted (in tasks.locked_by and task_attempts.run_id), then filters system.billing.usage to only those runs — scoping spend to the specific application.
Lakebase's autoscaling compute + storage separation makes Postgres-as-queue viable at scale. Unlike traditional Postgres deployments, compute scales with demand while storage remains durable and independent. OAuth-rotated auth eliminates static credentials.

Architecture¶

[Web App (FastAPI / Databricks Apps)]
       │ writes PDFs to UC Volumes + task rows to Lakebase
       v
[Lakebase Postgres]
  ├── tasks table (status, lease, priority, result)
  └── task_attempts table (run_id, mlflow_trace_id, cost)
       │
       v
[Orchestrator Daemon (Databricks Apps)]
  ├── dequeues with FOR UPDATE SKIP LOCKED
  ├── enforces rate limits (concurrency + token budget)
  ├── dispatches to Lakeflow Jobs
  └── operator dashboard (LISTEN/NOTIFY → SSE)
       │
       v
[AI Agents (Lakeflow Jobs)]
  ├── reads PDF from UC Volumes
  ├── processes via IDP + LLM/vision calls
  ├── writes results back to Lakebase
  └── fires idempotent webhook callback

Distributed Systems Patterns¶

Concurrent dequeue with priority¶

FOR UPDATE SKIP LOCKED + ORDER BY priority DESC, created_at
Concurrency-safe without external locks
Higher-priority work dispatched first, FIFO within each level

Lease-based crash recovery¶

Record lease_expires_at at dequeue time
Periodic sweeper re-enqueues expired leases
No heartbeat protocol, no external coordination service
Workers recovered within minutes of crash

Rate-limit-aware throttling¶

Three modes: concurrency cap, token budget, combined
Decision at dequeue time in same transaction as lock
Tasks that don't fit stay in queue for next cycle
No in-memory waiting queue or inter-replica coordination

Idempotent webhook callbacks¶

Accepts PROCESSING or ENQUEUED → transitions to terminal
Already-terminal states are no-ops
Eliminates double-billing under network retry

LISTEN/NOTIFY + SSE for real-time push¶

Postgres-native pub/sub, no Redis/WebSocket server
SSE fans events to browser within ~1s
Polling fallback at 10s for dropped streaming connections
UI indicator distinguishes live vs polling mode

Operational Numbers¶

Extraction time: hours → minutes (production at CLA)
Dashboard latency: ~1s via LISTEN/NOTIFY + SSE
Polling fallback: 10s interval
External systems eliminated: Kafka, Redis, Airflow, Temporal

Caveats¶

Tier-3 source (Databricks blog); describes a Databricks-native stack — patterns are general Postgres but the surrounding services are Databricks-specific.
"Token budget" throttling relies on projected token estimates, which may be inaccurate for variable-length documents.
Lease-sweeper interval determines worst-case task-recovery latency (not specified precisely in post).
No mention of WAL/vacuum pressure from high-throughput queue churn — a known concern for Postgres-as-queue at extreme scale.

Source¶

systems/lakebase — the Postgres-compatible OLTP database at center of this architecture
concepts/select-for-update-skip-locked — the core dequeue primitive
patterns/postgres-queue-on-same-database — the general pattern this article implements
concepts/leader-lease — related concept of time-bounded locks for distributed coordination
concepts/idempotent-operations — the design property that makes webhook callbacks safe
patterns/idempotency-key-header — related idempotency pattern for APIs
concepts/adaptive-rate-limiting — broader rate-limiting concept