Skip to content

Cloudflare — Cutting our inference bill by 68% by being obsessive about optimization

Summary

A direct follow-up to Cloudflare's 2026-04-20 AI code-review post. Between the April writeup ($1.19 average review) and late August, the average review cost had doubled to ~$2.46 with no architectural change — driven mostly by moving every reviewer onto Anthropic models (which raised per-job cost from $1.38 → $2.03 after an eval showed they performed better on the workload) and by the standards reviewer Codex landing on Claude Opus 4.7 ($0.25 → $0.88 per review). The prompt-cache hit rate rose from 85.7% (April) → 97% (September) but tokens-per-review more than tripled (0.91M → 2.99M), so cost climbed anyway. Mid-September Cloudflare pointed its own agents at the problem and cut the average review to $0.77 — down 68% from the $2.43 two-most-expensive-weeks baseline — while the median review time dropped from 4m 09s → 1m 32s (a side effect, not a goal). ~40% of the savings came from reviewer-code changes; ~60% from experimenting with which models run and how much effort they expend. The single biggest lever: checking what actually reached the provider on each session's first request and re-ordering messages so a shared prefix could be cached once and reused across every repository. The quality half of the post defends the cheaper reviews with two independent methods (acted-on-findings tracking + a blinded multi-model judge over 657 findings), concluding code-quality-at-high-effort finds about as many real problems as before with far less noise.

Key takeaways

  1. Cost doubled with no architectural change — model choice was the dominant driver. Average review cost rose $1.22 (Mar) → $1.44 (Jun) → $1.74 (Jul) → $2.32 (Aug). A small part was tier mix (59% of reviews in the most-expensive "full" tier in March → 65% in August, because better models let engineers write bigger changes). "Most of the cost increase actually came from model changes." At the start of August Cloudflare moved every reviewer onto Anthropic models after evals showed they performed best on this workload — raising per-job cost $1.38 → $2.03. (Source: sources/2026-10-19-cloudflare-cutting-inference-bill)

  2. A high cache hit rate does not bound cost when token volume grows. Prompt-cache hit rate went 85.7% (Apr) → 97% (Sep), but tokens per review more than tripled, 0.91M → 2.99M. "A high hit rate on three times as many tokens still costs a lot more, even with the low cache read cost." The canonical wiki datapoint that hit rate is necessary but not sufficient — the token-budget denominator matters as much as the ratio. (Source: sources/2026-10-19-cloudflare-cutting-inference-bill)

  3. Most of the spend was context the model had already seen — cache writes + reads dominated. Right before the optimization push, spend split: output tokens ~23%, cache writes ~43%, cache reads ~34%. The code-quality reviewer made 16.6 LLM calls per review on average, writing 72k tokens to cache and reading back 1.08M — "because each call sends the whole conversation as context." A session's cost grows with both the number of calls and the size of the context, and both had crept up. Two workstreams followed: make the first request of each session read from cache instead of writing to it, and make each session use fewer calls. (Source: sources/2026-10-19-cloudflare-cutting-inference-bill)

  4. Anthropic prompt caching only works on prefixes, and a prefix is only stored where a cache breakpoint ends it. Caching applies to messages at the start of the conversation (tool definitions + system prompt + opening messages), not mid-conversation. Write cost = 1.25× the normal input price; read cost = 0.1×; an entry expires after 5 minutes unless something reads it. The harness tools + system prompt + opening messages of a review session rarely change, making them an ideal prefix candidate. (Source: sources/2026-10-19-cloudflare-cutting-inference-bill)

  5. The biggest single win: share one cached prefix across every repository. Instrumentation (from Sep 14) recorded each reviewer's first-call token usage + a hash of the first system block exactly as sent to the provider. The first request found a cached prefix only 14–26% of the time, even though the first 11k–21k tokens are identical in every repo. Root cause: OpenCode v1 built one system block = agent prompt + an environment section (working directory + date) with the only cache breakpoint at the end of that block — so the shared part was never stored on its own and a review in one repo could never reuse what another repo wrote. Fix (Sep 16): add a marker to the end of every agent prompt + a plugin hook that splits the block in two at that marker, giving the first (shared) half its own breakpoint. To keep it shareable they removed everything repo-specific from in front of it — the list of available skills is disabled and tool descriptions can't change between jobs. (Source: sources/2026-10-19-cloudflare-cutting-inference-bill)

  6. Codex: move the reference index into the prompt instead of reading it with tool calls. The same day, the Codex (internal standards) reviewer's reference index — a table of internal standards, a skill file, and a list of reference files — was moved into its prompt. Until then every session had spent its first three turns reading those files with tool calls. Result: Codex LLM calls per run 15.9 → 9.6, cost per run $1.12 → $0.72. Together with the shared-prefix change, review cost dropped $2.47 → $2.02. (Source: sources/2026-10-19-cloudflare-cutting-inference-bill)

  7. Six "fewer calls, fewer dollars" code changes (Sep 21 + 23) removed ~$0.20/review combined. (a) Remove tools reviewers can't use — sub-reviewers could see the final-comment-posting tool and tried to call it in 57–76% of sessions, getting server-rejected and burning a recovery call (also ~2.9k fewer tool-definition tokens per request). (b) Read files in parallel — in 64% of code-quality turns the model made exactly one tool call (usually reading one file); now reviewers read shared context + all their patches (up to 150 KB) in their first response. (c) Skip reviewers on re-reviews — ~two-thirds of jobs are a 2nd+ review; on full-tier re-reviews AGENTS.md reviews are skipped 56% of the time, performance 23%, documentation 13%. (d) Stop after the review is posted — OpenCode previously passed the result back for one more turn to write a summary nobody saw (~$0.07/job). (e) Treat a slow reviewer differently from a broken provider — a reviewer that hit its time limit was being restarted like a provider failure; with longer time limits + lower effort, security-reviewer timeouts went 1.8% → ~0. (f) Cancel cancelled jobs — see takeaway 8. (Source: sources/2026-10-19-cloudflare-cutting-inference-bill)

  8. GitLab's Kubernetes executor doesn't signal a cancelled job's script — 358 of 446 sampled cancelled jobs kept running to completion. "GitLab's Kubernetes executor doesn't send a signal to a job's script when the job is cancelled." In a sample of 446 cancelled jobs, 358 kept running until the review was finished — paying full inference cost for work nobody would read. Fix: jobs poll their own status every 15 seconds and stop if cancelled. Canonical wiki datapoint on a CI-runtime abstraction leak costing real inference dollars. (Source: sources/2026-10-19-cloudflare-cutting-inference-bill)

  9. Shared-prefix fan-out across three reviewers, with one writing the cache and two reading it. The most involved code change: the code-quality, security, and performance reviewers now send identical first requests — same tools, a shared rules file, then the merge request with every patch included — with each reviewer's own instructions following after that shared block. One reviewer writes the cache, the other two read it. Because a cache entry can't be read until the response that wrote it has started streaming, and the three start at the same time, the 2nd and 3rd wait up to 20 seconds for the first's response to begin before sending their own first request. "It slows them down slightly and saves about nine cents per reviewer." An explicit latency-for-cost trade (concepts/batching-latency-tradeoff). (Source: sources/2026-10-19-cloudflare-cutting-inference-bill)

  10. Model + effort experimentation was the larger half of the saving ($1.06 of $1.67). Code changes accounted for $0.65 of the $1.67 drop; model/effort changes for $1.06 (each step compares weekday hours before vs after a change — explicitly not a controlled experiment). Moves: the coordinator had run three model/effort combos since August; on the same model, full-tier jobs on medium effort finished 20% sooner, so the other two were dropped. A randomized split showed Opus 5.5 at medium = no measurable change vs Opus 5 in outcomes/critical findings → switched; Opus 5.5 at high cost more than medium without meaningfully more findings. Opus 5.5 is cheaper per token than Opus 5, but price was only ~$0.20 of the drop — the rest came from sending/generating fewer tokens (since Sep: cache reads per review −82%, cache writes −54%, output −67%). (Source: sources/2026-10-19-cloudflare-cutting-inference-bill)

  11. Sonnet 5 → Sonnet 5.5 A/B: 47–62% cheaper per run at the same per-token price — pure token reduction. On Oct 3 every Sonnet reviewer was split 50/50 Sonnet 5 vs 5.5. After 2.5 days: 5.5 was 62% cheaper (code quality), 58% (performance), 47% (security). The two models cost the same per token, "so all of that comes from using fewer tokens." 5.5 posted about as many critical/warning findings on full-tier → all moved to 5.5 on Oct 5. On Sonnet 5.5 at medium effort the code-quality reviewer makes 1.7 LLM calls/run instead of 6.2 and more than half its runs finish in a single call (never opening another file); at high effort 3.4 calls/run. (Source: sources/2026-10-19-cloudflare-cutting-inference-bill)

  12. Fewer calls also meant faster reviews — the slowest tail improved most. Not a goal, a side effect. In the four weeks before Sep 14, 37.9% of reviews took >5 min; now just 0.7% do. Today's 99th percentile is faster than three-quarters of reviews were in April. Median full-tier 5m 16s → 1m 54s, lite 2m 58s → 1m 14s, trivial 77s → 30s. A good part of the work "just got us back to where we had been" — by August the median was 14% slower and p99 48% slower than the April numbers. (Source: sources/2026-10-19-cloudflare-cutting-inference-bill)

  13. Quality method 1 — acted-on-findings: cheaper reviews flagged fewer things, but each flag was far more likely to lead to a real edit. For samples of critical/warning findings, Cloudflare compared the reviewed commit with the author's next push and counted a finding "acted on" if the lines it pointed at changed (lines 30–40 away changed ~15% of the time in every period = coincidence baseline). Authors acted on 34% of code-quality serious findings before → 59% now. Fewer findings, but "findings that led to a change per 1,000 reviews is about the same" (132 vs 145, within margin). Across all four reviewers, serious findings that led to a change went 229 → 321 per 1,000 reviews. (Source: sources/2026-10-19-cloudflare-cutting-inference-bill)

  14. Quality method 2 — blinded cross-provider judges over 657 findings: noise and severity inflation fell, important findings held. Two models from different providers judged a blinded sample of 657 findings from the three Sonnet reviewers (each with the diff it referred to, without knowing which period it came from); disagreements went to a third model. Each finding labelled important / minor / duplicate / wrong / unverifiable + severity fair / too-high / too-low (concepts/llm-as-judge + patterns/stratified-evaluation-sampling + patterns/human-calibrated-llm-labeling). What disappeared was mostly minor findings + some duplicates + unverifiable-from-diff. Important findings per review 0.28 → 0.30 (intervals overlap). The clearest result: findings rated more severe than deserved fell 0.46 → 0.14 per review, a 70% drop — "much of the drop in critical findings is the reviewer no longer calling minor things critical." (Source: sources/2026-10-19-cloudflare-cutting-inference-bill)

  15. The answer depends on the reviewer and the effort level. Code-quality at high effort: what disappeared was noise + inflated severity, still finds about as many real problems → moving code-quality to high (randomized split: high costs 61% more per run and posts 58% more critical+warning findings; medium led to ~¼ as many acted-on findings as high). Medium effort genuinely lost real findings: performance lost ~64% of important issues per review (≈ one fewer important finding every 24 full-tier reviews); security fell ~a quarter (not statistically significant). Both stay at medium for now; testing them at high is next. Reliability held: reviews ending without a verdict 0.79% → 0.27%; security-reviewer retries 3.5% → 0.04%. (Source: sources/2026-10-19-cloudflare-cutting-inference-bill)

  16. OpenCode v1 → v2 was validated by comparative shadow mode before cutover. The reviewer now runs on OpenCode v2 (a ground-up rewrite of v1). Before switching, Cloudflare ran every review in every canary repo on both v1 and v2 in comparative shadow mode; the two agreed on the approval decision in 98% of merge requests → everyone moved over in one go in early October and v1 code was removed. On two days with the same models, full-tier reviews on v2 finish ~13% faster. Cloudflare is also experimenting with running the reviewer solely on Durable Objects using pi-durable. (Source: sources/2026-10-19-cloudflare-cutting-inference-bill)

  17. Net result + the two things to check first if you run agents at volume. Average review $2.43 → $0.77; median review 4m 09s → 1m 32s. The two first checks Cloudflare recommends: "whether each session's first request actually reads from the cache, and whether your reviewers still find real problems at the effort level you're paying for." (Source: sources/2026-10-19-cloudflare-cutting-inference-bill)

Systems / concepts / patterns extracted

Operational numbers

Metric Before After
Average review cost $2.43 (2-week peak) / $2.46 (late Aug) $0.77 (−68%)
Median review time 4m 09s 1m 32s
Prompt-cache hit rate 85.7% (Apr) 97% (Sep)
Tokens per review 0.91M (Apr) 2.99M (Sep)
First-request prefix-cache hit 14–26% (shared prefix after Sep 16)
Shared identical prompt prefix 11k–21k tokens —
Spend split (pre-optimization) output 23% / cache-write 43% / cache-read 34% —
Code-quality calls/review 16.6 (Anthropic) → 6.2 (Sonnet 5 med) 1.7 (Sonnet 5.5 med) / 3.4 (high)
Codex calls/run · cost/run 15.9 · $1.12 9.6 · $0.72
Reviews taking >5 min 37.9% (pre-Sep-14) 0.7%
Median full-tier / lite / trivial 5m16s / 2m58s / 77s 1m54s / 1m14s / 30s
Cache reads / writes / output per review (since Sep) — −82% / −54% / −67%
Sonnet 5.5 vs 5 cost/run — −62% CQ / −58% perf / −47% sec
Acted-on serious findings (code quality) 34% 59%
Serious findings → change / 1,000 reviews (all 4) 229 321
Over-severe findings per review 0.46 0.14 (−70%)
Important findings per review (judges) 0.28 0.30
Cancelled jobs still running (sample) 358 / 446 polls own status every 15s
Reviews ending without verdict 0.79% 0.27%
Security-reviewer retries 3.5% 0.04%
OpenCode v1↔v2 approval agreement 98% —
v2 full-tier speedup (same models) — ~13% faster
Savings attribution code ~40% / model+effort ~60% ($0.65 vs $1.06 of $1.67) —

Caveats

  • Not a controlled experiment. "Each step compares the weekday hours before a change with the hours after it, so this isn't a statistically sound controlled experiment." The $0.65 (code) vs $1.06 (model/effort) attribution is directional.
  • Quality-eval caveats stated by Cloudflare: an author changing flagged lines shows they did something, not that the finding was right; the judges are models too; only 35–66 findings in each group; the "now" window is only a day long. The two methods agree with each other and neither suggests code-quality finds fewer real problems.
  • Medium effort demonstrably loses real findings for performance and (less certainly) security — the cost figures in the post assume code-quality moved to high effort.
  • Cost figures are relative, not absolute fleet spend; model names (Opus 4.7/5/5.5, Sonnet 5/5.5) are as stated in the post.

Source

Last updated · 854 distilled / 2,448 read