PATTERN Cited by 1 source

LLM Judge as Inline Pipeline Stage¶

LLM Judge as Inline Pipeline Stage is the pattern of embedding an LLM-as-judge inside the data pipeline as a first-class stage — every record produced by an upstream LLM step flows through the judge in the same run, gets a categorical rating + written justification, and gets routed (kept / flagged for manual review / discarded) before it reaches downstream consumers.

The contrast with classical LLM-as-judge usage: it is not a post-hoc audit on a sampled batch. It is not a leaderboard evaluation harness. It is the gating mechanism on every output, on every run, in production.

Problem¶

LLM pipeline outputs are non-deterministic and silently variable. Without inline quality scoring you face a binary choice:

Trust everything. Downstream consumers see all outputs, including the bad ones. Quality issues only surface when a human notices.
Audit afterwards. Sample-based post-hoc evaluation catches some bad outputs but the bad outputs already shipped. By the time you flag them they're already in the materialised dataset.

Neither option is acceptable when the pipeline output feeds a production system (groundwater predictions, search index, triage queue) where bad records have downstream cost.

Solution¶

Add a judge stage to the pipeline DAG between the producer LLM stage and the downstream consumers:

Upstream LLM stage
   produces classification / extracted record
            │
            ▼
Judge LLM stage (inline)
   scores against rubric (accuracy / completeness / consistency)
   emits: {rating: excellent|good|fair|poor, justification: "..."}
            │
            ├── rating ∈ {excellent, good} ──> downstream consumers
            └── rating ∈ {fair, poor}      ──> manual review queue

In the MapAid groundwater pipeline:

"A separate AI model, also called via AI Functions, acts as a judge: scoring every classification on a structured rubric covering accuracy, completeness, and consistency. For each document, the evaluator compares the assigned Dewey Decimal codes and geographic tags against the sampled page content, checking whether the classifications are supported by what the model actually observed. Each evaluation produces both a categorical rating (excellent, good, fair, or poor) and a written justification explaining the score, creating an auditable trail for every decision the pipeline makes. Documents scoring below a confidence threshold are flagged for manual review, directing limited human effort to the cases where it matters most. In the first full run, only a small fraction of classifications required human attention."

The pattern's three load-bearing properties are:

Every record judged. Not sample-based; not post-hoc.
Rating + written justification. The justification is the audit trail. "Why was this rated fair?" is answered by reading the justification column.
Threshold-based routing into manual review. The pipeline acts on the score by gating documents into a review queue — the judge isn't observability, it's a control gate.

Mechanics¶

Same inference primitive for producer + judge. In the MapAid pipeline both are ai_query calls; only the prompt + target model differ. This keeps the pipeline shape uniform — no special "judge service."
Different model. "A separate AI model… acts as a judge." Using a different model than the producer reduces correlated errors (the judge is less likely to make the same mistake the producer did).
Structured rubric. Accuracy / completeness / consistency in the MapAid case. The rubric is part of the call contract — same JSON schema every time. See concepts/schema-constrained-llm-output.
Review queue is a Delta table. Sub-threshold rows are materialised in a separate Delta table for human reviewers; the main pipeline doesn't block.
Confidence threshold is a tunable parameter. Tighten it and more documents go to manual review (higher quality, more human cost). Loosen it and fewer (lower quality bar, less human cost).

What inline judging buys you over post-hoc¶

Bad outputs never reach downstream consumers. Materialised outputs in the trust path were all judge-cleared.
Manual review is targeted. "Limited human effort directed to the cases where it matters most." No one reviews 5,570 pages by hand — they review the ~5% the judge wasn't confident about.
Audit trail is a column. Compliance / explainability ask "why did the pipeline classify this document as foo?" — the answer is the judge's justification stored alongside the classification.
Quality regression is observable. If model drift starts producing worse classifications, the judge's excellent/good rate drops. That's a metric you can alert on.

Why the judge needs its own design¶

Judge bias is real. A judge LLM rewards verbose / confident- sounding wrong answers. The rubric must be specific enough to resist this. The MapAid rubric ("checking whether the classifications are supported by what the model actually observed") anchors on observable evidence.
Judge model selection matters. Use a different model from the producer to break correlated errors. Don't use the cheapest model in your stack as judge — judge mistakes propagate to the manual-review filter.
Judge versions must be snapshot. When the judge model is upgraded, scores aren't directly comparable to historical scores. Store the judge version alongside each rating.

When to use¶

Production LLM pipelines whose outputs gate downstream actions (search index, triage queue, automated decision).
Pipelines where bad outputs cost more than the inference cost of judging.
Pipelines where a manual-review channel exists and you need to target it efficiently.
Pipelines where explainability is required (regulatory, scientific, humanitarian).

When not to use¶

Pipelines where outputs are advisory and a human reviews each one anyway. The judge is redundant.
Pipelines where the judge inference cost dominates the producer cost. Sample-based audit may be more economical.
Pipelines where no review channel exists. A judge that flags bad outputs but has nowhere to send them is just expensive observability.

Tradeoffs¶

2× inference cost at minimum (producer + judge per record).
Judge prompt is now part of the production surface — needs versioning + deployment discipline.
Threshold tuning becomes a product decision. Tight = high quality, expensive humans. Loose = cheap, lower quality ceiling.

Seen in¶

sources/2026-05-11-databricks-unlocking-the-archives — canonical wiki instance. Inline judge over every classification (5,570 pages aggregated to 654 documents); rating + written justification; sub-threshold to manual review; 95% rated excellent/good in first full run; "only a small fraction" required manual review.