Skip to content

DATABRICKS 2026-10-27

Read original ↗

From spot fixes to measured gains: improving document extraction accuracy with ai_extract and MLflow Prompt Registry

Summary

A Databricks field-engineering post about change management for a production LLM extraction pipeline. The project extracted ~40 structured fields from partner-submitted order PDFs across dozens of layouts; human reviewers corrected and approved every extracted value before it went downstream. The pipeline was a medallion flow on Lakeflow SDP: object storage → Auto Loader → Bronze (ai_parse_document) → Silver (ai_extract, a Databricks AI Function) → Gold materialized view → synced to Lakebase, where a Databricks App lets reviewers correct/approve, and Lakebase CDC returns approvals to the lakehouse. The trigger for the post: a one-field "spot fix" to the schema halved errors on the revision field but raised total errors across all fields by 9% — the model reads the schema as a whole, so a one-field edit is not a one-field experiment. The fix is an engineering workflow with three parts: (1) pick the right lever (schema vs global prompt vs precision mode vs deterministic post-processing), (2) version prompt and schema together in MLflow Prompt Registry and deploy/roll back by moving aliases (@prod, @candidate, @scr_*) rather than redeploying pipeline code, and (3) gate promotion on a frozen, human-reviewed ground-truth cohort scored on accuracy, coverage, and run-to-run variance (a 2σ noise floor), collapsed into a verdict (BETTER / SUSPECT / UNSTABLE / INCONCLUSIVE / WORSE) — with the final alias move kept human-controlled.

Key takeaways

  1. Local improvement ≠ system improvement. The revision schema fix cut revision errors −49% but total errors +9%: the model began inventing values in unrelated line-item fields (schedule details). Editing one field's description perturbs extraction of the whole schema, so every change must be scored across all fields. (Source: this page)
  2. Put each rule where it belongs. Field rules (enums, numeric types, nested arrays, "preserve leading zeros", "exclude ORIGINAL/NEW/REVISED", "return null if absent") go in the JSON schema; document-level reasoning (domain context, header-over-footer source priority, cross-field consistency, row→array assembly, FORBIDDEN-SOURCES regions) goes in the global prompt. Rule of thumb: "If the rule defines one field, put it in the schema. If it defines how to interpret the document, put it in the prompt." Restructuring this way halved the prompt's length and cut total errors −16%. (Source: this page)
  3. Use the model to interpret, code to normalize. Casing, separator removal, zero-padding, and value mappings (TELEVISION → TV) belong in SQL downstream, not in the LLM — a direct instance of patterns/deterministic-tool-vs-llm-judgment. (Source: this page)
  4. Diagnose meaning vs structure before choosing the lever. If the model misunderstands what a field means, change prompt/schema; if it understands but struggles with the data structure (repeated line items, complex tables, multi-page layouts), test precision mode (ai_extract v2.1). Precision mode alone: total errors −24% (mostly by no longer inventing values in repeated line items, i.e. less over-extraction) but only −10% on revision, a definition problem only the schema rule fixed. The two stack: restructured prompt+schema + precision mode = −32% total, −64% on revision. (Source: this page)
  5. Citations separate extraction errors from labeling disagreements. Field-level bounding boxes mapped to source element spans show where the model read a value: wrong region / unsupported value → fix schema or prompt; right region but different reviewer convention → fix reviewer guidelines. Confidence scores are a ranking signal for review order, not a quality gate — "a high-confidence result can still be wrong." (Source: this page)
  6. Version prompt and schema together, outside pipeline code. Hardcoded prompts mean a wording change requires a code deployment, history is hard to reconstruct, and there's no evidence the new version is better. Store both in MLflow Prompt Registry — "version only the prompt, and half of your extraction logic is still traveling as an untracked code change." The Silver step resolves the active versions by alias at pipeline initialization; promotion = mlflow.genai.set_prompt_alias moving @prod; rollback = restoring both aliases to the previously recorded versions (old versions are never deleted). This is patterns/fast-rollback via an indirection pointer, decoupled from code deploys. (Source: this page)
  7. Reviewer approvals become reusable ground truth. A ground-truth view pairs approved labels with source docs and predictions; the evaluation cohort is a frozen set of several hundred reviewer-approved PDFs spanning layouts. Keep documents and metric definitions fixed across comparisons, else a score delta may reflect a different test set or scoring rule. This closes the human-in-the-loop review into an evaluation loop. (Source: this page)
  8. Measure the noise floor. Run the production config 3–5 times on the frozen set; its σ is the noise floor. A policy may require a candidate's gain to exceed 2σ; anything inside the band is INCONCLUSIVE, not a win. (Source: this page)
  9. Accuracy without coverage lies. Scoring only fields where both model and reviewer supplied a value lets a candidate that blanks hard fields look more accurate (misses leave the denominator). Measured that way the revision fix looked like <3 pp; counting all errors (incl. correctly returning blank) it halved revision errors. Define coverage explicitly (share of non-null ground-truth fields the model predicts) and investigate if accuracy rises while coverage falls — a denominator trap in the concepts/benchmark-methodology-bias family. (Source: this page)
  10. Verdicts inform, humans decide. Harness output collapses to BETTER / SUSPECT (accuracy up, coverage down) / UNSTABLE / INCONCLUSIVE / WORSE, but a severe single-field regression (a candidate that filled a yes/no flag on nearly every document when reviewers had marked it on a handful) can block an otherwise-acceptable candidate; conversely an INCONCLUSIVE change may be promoted for a business convention, with a recorded override reason. Each promotion records candidate version, metrics, author, timestamp. (Source: this page)

Architecture

object storage ─▶ Auto Loader ─▶ Lakeflow SDP
   Bronze: ai_parse_document
   Silver: ai_extract(parsed_doc, schema@alias, prompt@alias)   ◀── MLflow Prompt Registry
   Gold:   materialized view                                          (@prod / @candidate / @scr_*)
        │                                                                    ▲
        ▼ sync                                                               │ owner moves @prod
   Lakebase ◀─▶ Databricks App (reviewers correct / approve)                 │
        │ CDC                                                                │
        ▼                                                                    │
   ground-truth view (approved labels + docs + predictions) ─▶ eval harness ─┘
        frozen cohort · N repeated runs · accuracy / coverage / σ · verdict

Evaluation sequence (Figure 3): freeze test set → measure production baseline (repeated) → register candidate on a screening alias → run repeated evaluations → compare accuracy, coverage, stability → verdict → human promotion. Every MLflow run logs prompt version, schema version, and engine, so each score traces to the exact configuration.

Operational numbers

Change (3 runs each, vs original) revision errors All-field errors Verdict
revision schema fix only −49% +9% TARGET_IMPROVED_WITH_REGRESSION
Restructured prompt + schema −57% −16% BETTER
Precision mode only −10% −24% BETTER
Restructured prompt + schema + precision mode −64% −32% BETTER
  • ~40 extracted fields; dozens of PDF layouts; eval cohort of several hundred PDFs.
  • Restructured prompt < ½ original length; estimate errors −30%; two fields worsened and went onto the next round's target list.
  • Precision mode on top of restructured config removed a further 19% of remaining errors.
  • Noise floor: 3–5 baseline runs; example promotion bar >2σ.

Systems / concepts / patterns extracted

Caveats

  • Tier-3 vendor post; the workflow doubles as promotion for Genie Code, ai_extract v2.1 and MLflow Prompt Registry. All numbers are from a single unnamed customer project, self-reported, relative percentages only (no absolute error rates, no σ values disclosed).
  • Each candidate scored only 3 times — at the low end of the post's own 3–5-run noise-floor recommendation.
  • Alias resolution happens at pipeline initialization; the post does not discuss in-flight runs during an alias move, or atomicity when prompt and schema aliases must move together (two separate alias updates).
  • Ground truth is reviewer-approved output of the same system, so it can inherit anchoring bias toward model predictions; the post acknowledges labeling disagreements but not this bias.

Source

Last updated · 854 distilled / 2,448 read