Skip to content

DATABRICKS

Read original ↗

Provisioning for the Agentic Era: How Databricks Built a Self-Serve Infrastructure Vending Machine

Summary

Databricks' field engineering organisation (7,000+ GTM, significant field-engineering portion) outgrew shared workspaces that worked at 1,500 people. The Field Engineering Vending Machine (FEVM) replaces shared environments with isolated, just-in-time, use-case-based provisioning: engineers describe what they need (demo for a financial services customer, hackathon session, support reproduction), and FEVM provisions a configured environment in minutes with automatic TTL-based cleanup. Built as a Databricks App (React + Python backend) with Terraform for multi-cloud resource provisioning, Lakebase for state/configuration tracking, and an MCP interface enabling agent-first workflows.

Key Takeaways

  1. Shared-workspace coordination doesn't scale. At 7,000+ GTM users with most needing admin access, shared environments create interference, platform-limit contention (catalogs, Lakebase instances, concurrent workloads), cost-attribution opacity, and observability gaps.

  2. Use-case-based provisioning over resource-based provisioning. Engineers don't request "a workspace" — they describe what they're trying to accomplish in plain language, and the system maps intent to infrastructure configuration via a template catalog. This abstraction is the core design principle.

  3. Agent-first API via MCP. The MCP at the core means chat, external services, and coding agents all access the same provisioning surface. Adding a .md skill file lets Claude (or similar) provision environments via plain-English commands on the command line — "ask for a new environment… and log in minutes later."

  4. Multi-step agent workflows become trivial. An agent can spin up a workspace, deploy DABs, upload S3 data, and run a hydration script — hyper-personalised setup that wouldn't be possible without a strong agent framework, made trivial by Databricks Apps.

  5. Template catalog for governed provisioning. Templates are vetted, hardened, and tested for security — no ad-hoc provisioning that could expose threat vectors. New templates go through a vetting pipeline.

  6. TTL-based resource lifecycle with notification transparency. Builder environments live 90 days by default with extensions. Every lifecycle event (provisioning, approaching expiration, deletion) sends a Slack notification. "Transparency was a core design goal from day one."

  7. Resource-level lifecycle management decouples catalogs from workspaces. Standalone catalogs have independent lifecycles — delete a workspace and the catalog persists; spin up a new workspace in the same region and it reattaches. Critical because Unity Catalog and Lakebase have hard platform limits.

  8. Scale validated at burst. BuildCon: ~1,200 provisioning requests in a single day. Ongoing: 5,000+ active users, 2,600+ active deployments across 3 clouds, no scalability issues hit.

  9. Rearchitected twice. Database schema redesign, frontend rebuild, state-management rethink — AI accelerated the coding but architecture decisions stayed human-driven.

  10. Vision: agent-first field engineering. An engineer describes a customer situation → agent provisions environment → retrieves account context → reproduces issue or builds demo → returns summary. FEVM is the infrastructure layer.

Architecture

  • Frontend: React app deployed via Databricks Apps
  • Backend: Python, workspace-resident service principal auth
  • Provisioning: Terraform templates fetched and fed to a Git Runner per request
  • State DB: Lakebase — tracks every resource (what, who, what for, when it expires)
  • Multi-cloud: AWS, Azure, GCP provisioning
  • Agent interface: MCP server as single point of control for chat / external services / agents
  • Admin layer: global configuration, workspace limits by cloud, audit of active deployments

Operational Numbers

Metric Value
Active users 5,000+
Active deployments 2,600+ across 3 clouds
Peak single-day requests ~1,200 (BuildCon)
Default environment TTL 90 days (extensions available)
GTM org size 7,000+

Systems & Concepts Extracted

Caveats

  • Tier-3 source; lighter on distributed-systems internals than typical Tier-1 ingests
  • No architecture diagrams with internal component detail (one high-level diagram referenced but not deeply described)
  • Terraform template structure, Git Runner mechanics, and Lakebase schema not disclosed
  • Security-exception framework mentioned but not detailed
  • No latency numbers for provisioning (only "minutes" stated)
  • No cost data or resource-utilisation efficiency metrics
  • "Rearchitected twice" claim without architectural-before/after detail
  • BuildCon is an internal event; 1,200/day may not represent sustained external load

Source

Last updated · 593 distilled / 1,805 read