Skip to content

DATABRICKS 2026-10-22 Tier 3

Read original ↗

The AI Bill is coming

A conversation between Catherine Brown and Patrick (who runs Databricks' AI products and championed internal AI adoption) about what happened when Databricks went all-in on AI coding agents: costs that initially spiralled on an exponential curve, the levers that brought unit costs down by as much as 90%, and the thesis that every engineering org needs a control layer between its developers and the models. That control layer began as an internal fix for Databricks' own AI bill and became the inspiration for Unity Gateway. It is an interview-format restatement and sharpening of Databricks' earlier Managing AI Coding Costs at Scale playbook and the Day-1 model-rollout post, with a few fresh framings.

Summary

Engineering output is the business bottleneck, so productivity tooling is worth a lot — Databricks reports close to doubling output with the same team. But coding agents are consumption-priced, not per-seat, so a single user can "run a little loop on the most expensive model and go to lunch" and spend an unbounded amount. When Databricks removed every barrier to drive adoption, cost grew exponentially while revenue didn't, "threatening to reverse the efficiency gains." The fix is not a hard cap (which cuts off your best users mid-task and pushes them to personal accounts you can't see) but a control layer — Unity Gateway — that applies graduated friction (warn → approve → downshift to a cheaper model → cut off only as a last resort), gives developers spend visibility, moves traffic onto newer, cheaper models as they ship, and routes each request to the cheapest capable model. Underneath all of it: make AI spend something you can see and steer by funnelling every tool's traffic through one gateway.

Key takeaways

  1. Consumption pricing makes a single developer an unbounded cost. Coding tools are consumption-priced now, not per-seat; a user can loop the most expensive model indefinitely. "If your costs grow exponentially and your revenue doesn't, at some point you have a problem." (Source: The AI Bill is coming) This is the cost pathology concepts/token-overhead and concepts/efficiency-frontier exist to manage.
  2. "Tokenmaxxing → ValueMaxxing." The failure mode is maximising token consumption for its own sake; the goal is value per dollar. Databricks reports token spend per head roughly flat even as total usage and value keep rising — the signature of a working control layer.
  3. Hard spend caps are the wrong mechanism; graduated friction is right. A hard stop cuts someone off mid-task (losing hours of flow) and the people who hit it are often the best users, who then move to personal accounts you can't observe. Unity Gateway instead ramps friction: a warning, then an approval, then a downshift to a cheaper model so work continues; cutting off is the last resort. A graceful-degradation posture applied to cost, not availability.
  4. Spend visibility is itself a ~10% lever. Sending every developer a regular breakdown of their own spend with savings suggestions is "worth about 10%." One peer company "puts the model and the cost right in every pull request" — public shaming for using an expensive model on a small task. Cost observability changes behaviour before any automated control does.
  5. The biggest lever is boring: move traffic to newer, cheaper models. "Better, cheaper models keep coming out, and you move traffic to them." ~5 models ship weekly across vendors; roughly one every week or two advances the price/performance efficiency frontier. Because everything goes through the Gateway, Databricks switches models in one place with no developer change — "50% or more on its own." Add routing + spend visibility + token-overhead cuts and unit costs are down as much as 90%. (Source: The AI Bill is coming)
  6. A 100× cost drop in under a year. A recent cheap model "roughly matches what a frontier model could do about nine months ago, at around 1% of the cost per task" — a 100× drop in under a year (preliminary, but directional). The catch: "you have to actually move" — an unused new model is "a tree falling in the woods."
  7. A model's token price is a bad predictor of its task cost — benchmark on your own code. Databricks benchmarks agents against its own multi-million-line codebase and grades by running the tests, not by LLM-as-judge ("that tends to reward sounding right over being right"). Surprises: open-source GLM 5.2 tied Opus 4.8 on quality at ~⅓ less per task; a model 1.7× cheaper per token cost more per task because it burned more tokens — the public-benchmarks-don't-predict-your-workload lesson and the "cheaper per token ≠ cheaper per task" rule.
  8. The harness matters as much as the model. "Same model, different harness, more than 2× difference in cost." Just tuning harness + caching settings saved "close to 50%" — the concepts/token-overhead lever, where the meta-harness is the switch point.
  9. Frontier models are for frontier problems (Lamborghini / Honda / bicycle). Most coding work doesn't need the biggest model. Rolling "Astra" out to all ~3,500 engineers raised spend ~60% with no obvious gain on everyday tasks — a lot of it waste. The fix: give the premium model its own budget in Unity Gateway, reserved for hard problems — the quality-frontier budget of patterns/experimental-tier-model-promotion.
  10. Routing is the new AI battleground, and it's an infrastructure problem. A budget can't decide which model handles a request — that's routing. Databricks' Smart Router cuts average task cost >30% at roughly the quality of the most expensive model, and removes the ask-the-user-which-model friction. Routing models must be "tiny and fast," it's "hard to do well," and "there's real IP in getting it right" — as model margins thin, routing is where the competition moves.
  11. One advice to a CTO: figure out where it's going, and don't slow down. Most companies can't say what they spend on AI by team or model. "Put a gateway in front of your models so all the traffic goes through one place, and you can see it. Everything else gets easier after that." The productivity gains are real and the bill compounds — waiting makes it bigger.

Systems / concepts / patterns extracted

Operational numbers

  • ~2× engineering output with the same team (aggregate; teams that rebuilt process around AI go faster).
  • Token spend per head roughly flat while usage and value rise.
  • Spend-visibility email: "worth about 10%."
  • Moving traffic to newer/cheaper models: "50% or more on its own."
  • Harness + cache tuning: "close to 50%"; **same model, different harness =

    2× cost difference.**

  • Combined unit-cost reduction: up to 90%.
  • 100× cost-per-task drop in under a year (cheap model ≈ 9-month-old frontier at ~1% cost per task; preliminary).
  • GLM 5.2 tied Opus 4.8 at ~⅓ less per task; a 1.7× cheaper-per-token model cost more per task.
  • "Astra" rolled to ~3,500 engineers → +~60% spend, little everyday gain.
  • Smart Router: >30% average task-cost reduction at ~top-model quality.

Caveats

Source

Last updated · 854 distilled / 2,448 read