Skip to content

DATABRICKS 2026-10-22 Tier 3

Read original ↗

The new economics of AI cost management

An interview with Kevin Stumpf (Senior Staff Engineer at Databricks, Technical Lead for Unity Gateway) framing AI cost management as a system to manage, not a line item to minimise. The through-line: the industry is moving from "tokenmaxxing" to "valuemaxxing" — optimising business value per dollar rather than raw token volume — and the enterprises that build the cost-management discipline first are the ones that can scale AI without spend outrunning revenue. It is a sibling interview to Databricks' earlier The AI Bill is coming (sources/2026-10-22-databricks-the-ai-bill-is-coming), covering the same levers (attribution, routing, guardrails, tracing) from a slightly different engineer's angle, with Unity Gateway as the centralised control point throughout.

Summary

AI cost management is defined as "measuring, attributing, controlling and optimizing AI spend across models, agents, applications and teams," with the goal of maximising value per dollar rather than minimising tokens. The interview walks each discipline in turn: cost attribution (granular enough to tie spend to a user / team / project / org-chart node, extended arbitrarily by request-level tags so a tag can represent a PR, a feature, or an external customer); taming context bloat by not generating it in the first place — letting agents pull context on demand through tools / MCP / skills rather than pushing everything up front (the "90% certain it's needed" rule); dynamic model and harness routing to the lowest-cost option that clears the quality bar (only ~5–10% of tasks genuinely need a frontier model); guardrails on spend that ramp responses (alert → hard cap → capacity reduction) rather than a single hard stop; measuring ROI by flowing session/request tags through to a cost dashboard so a PR can be costed and eventually tied to a feature and its revenue; and end-to-end tracing that collects the full "exhaust" of agent behaviour for later insight. The architectural prerequisite under all of it: every AI interaction must flow through one centralised system (an AI gateway), and adoption must be universal — "visibility only works if adoption is universal, not optional."

Key takeaways

  1. Tokenmaxxing → valuemaxxing. Early leaderboards rewarded raw token volume and "got gamed" (spin up a dumb agent that asks the weather in a loop). The right metric is impact generated per dollar spent — ROI — which is "a much harder thing to quantify than raw token counts, but it's the number that actually matters." (Source: The new economics of AI cost management)
  2. Cost attribution answers two questions: where did spend come from, and what did I get for it. Databricks attributes out of the box by user, group, agent, AI gateway endpoint, model, provider and time period, and allows hyper-customisation via request-level tags — a tag can represent a PR, a feature, or an external customer, and once every request carries it you can aggregate spend against it however you need. "You can't manage what you can't measure." This is the attribution substrate of concepts/centralized-ai-governance.
  3. Sentiment flipped in ~6 months. The top concern went from "falling behind on adoption" to "costs are growing faster than revenue." It's now a "boardroom-level conversation at nearly every organization," including internally at Databricks.
  4. Context bloat: don't generate it, rather than trim/cache it afterward. Bloat (throwing more info at an agent than it needs) burns tokens and distracts the agent. The best fix is pull-on-demand — let the agent fetch relevant context through tools, MCP servers, or skills rather than pushing everything up front. Rule of thumb: "only push context to an agent by default if you're 90% certain or more that it's genuinely important." A context-engineering / token-overhead discipline. (Source: The new economics of AI cost management)
  5. Model selection should be dynamic, not pick-once. Roughly 5–10% of tasks genuinely need a frontier model (high stakes or genuinely hard); the other ~90% are solvable by cheaper/open models. Left to their own incentives, employees default to the most expensive model because "nobody is watching what it costs them individually." Smart Routing inserts a layer that assesses task complexity in real time and picks a frontier vs lower-cost model to hold quality at lower cost — concepts/model-first-routing.
  6. Harness routing is the next layer beyond model routing. "The exact same model, run through a different harness, can produce meaningfully different costs" — the harness governs when the context window is compacted, what rules are followed, what business logic runs, all of which drive token spend. Databricks enables harness routing through Unity Gateway and Omnigent, its meta-harness — the "crawl-walk-run" step after model routing. (systems/omnigent, concepts/token-overhead.)
  7. Guardrails on spend should ramp, not hard-stop. Everyone should have a hard spend cap set above a typical power user's peak (e.g. ~$500/day cap if a power user spends ~$300/day) to catch runaway loops without clipping legitimate high-usage days. Beyond that, differentiated budgets by team or project weighing value vs headroom, and budgets that trigger graded responses — alert, hard cap, or reduce available model capacity (e.g. cut default frontier access at 70% budget burn and route the rest to open-source models). Core principles: prevent runaway spend, set differentiated limits, alert before you cap, and build intelligent capacity reduction rather than an abrupt shutoff — a graceful-degradation posture on cost.
  8. Measuring ROI is the "holy grail," and tagging is what makes it possible. Session/request-level tags flow attribution back into a cost dashboard, so a given PR can be shown to cost e.g. $300 to complete (Databricks does this internally today). The in-progress step is connecting PR → feature → a measurable outcome (revenue or pipeline it generated), turning "what did AI cost" into "what did AI return." Databricks is building this into Unity Gateway so it's available out of the box rather than hand-constructed.
  9. Spend visibility requires a centralised system, and universal adoption. Good visibility "requires that every AI interaction flows through a centralized system" — an AI gateway as the control point across models / agents / tools / teams, where you can observe usage, attribute cost, enforce budgets and optimise spend in one place. But "visibility only works if adoption is universal, not optional" — you need a way to enforce that everyone routes through it. patterns/central-proxy-choke-point + concepts/centralized-ai-governance.
  10. Cost optimisation can't outrun quality — you need a feedback mechanism. Subjective feedback (someone says they can't do the job within budget) is fast but easy to dismiss; objective feedback from automated evaluations is harder to set up but far more reliable. For coding agents, define a proxy quality metric — e.g. merged PRs that don't get rolled back within two weeks — then run experiments (halve a budget and watch the gold-star metric move, hold, or even improve since "constraints sometimes breed creativity"), with parallel experiments across teams or time-bound comparisons to infer whether a budget creates or destroys value rather than guessing.
  11. End-to-end tracing collects the "full exhaust" of agent behaviour. Traces capture what agents are thinking, which tools they call, what data they pull, whether prompts are good or bad — raw data you can mine for abuse, waste, security leaks, or patterns you never thought to look for. The analogy: structured DBs → data lakes → "data swamps" → usable data lakes again; agent traces are the same unstructured exhaust, now minable because "LLMs are flexible enough to process it." The observability / telemetry-to-lakehouse substrate.
  12. Agent fleets multiply the equation, not change it. "Agent swarms are going to be the new reality," and every agent spends tokens that must be attributed, monitored and managed — coding agents dominate today but it will expand to "agents calling other agents calling other agents." The advice: put the cost-visibility and management foundation in place now, "before the number of agents in production grows," or you'll build it under pressure. "You can't manage what you don't measure."

Systems / concepts / patterns extracted

Operational numbers

  • ~5–10% of tasks genuinely need a frontier model; the other ~90% are solvable by cheaper/open models.
  • Same model, different harness → meaningfully different cost (restated here qualitatively; the companion AI Bill interview puts it at >2×).
  • Guardrail example: power user ~$300/day → set the hard cap ~$500/day.
  • Capacity-reduction trigger example: cut default frontier access at 70% budget burn, route the remainder to open-source models.
  • ROI example: a given PR shown to cost ~$300 to complete (internal today).
  • Cost-quality proxy metric example: merged PRs not rolled back within 2 weeks.

Caveats

Source

Last updated · 854 distilled / 2,448 read