Skip to content

SYSTEM Cited by 1 source

LACE (Instacart LLM-Assisted Chatbot Evaluation)

LACE is Instacart's internal offline-evaluation framework for its AI-powered customer-support chatbot. It scores every evaluated chat session — a full multi-turn customer ↔ chatbot conversation — against a binary (True/False) rubric across five quality dimensions, using a multi-agent debate evaluation engine for context-sensitive criteria. LACE bootstraps and regression-tests itself against a human-rated ground-truth set, and in production feeds dashboards + Instacart's experimentation platform for continuous chatbot improvement (Source: sources/2025-06-11-instacart-turbocharging-customer-support-chatbot-development-with-llm).

Why it exists

A chatbot is only as good as the evaluation that measures it. Moving from human-agent to AI-agent support trades agent cost and queue latency for quality-signal loss — the chatbot can silently degrade on rare issue types, contextual misunderstandings, or policy drift. Human review doesn't scale to continuous monitoring of chatbot interactions across the whole traffic mix. LACE is the automated-judge that substitutes.

Architecture

LACE has three load-bearing design moves:

1. Binary rubric over five dimensions

Each chat session is evaluated against True/False criteria grouped under five dimensions:

  1. Query Understanding — did the chatbot correctly interpret what the user was asking?
  2. Answer Correctness — contextual relevancy, factual correctness, consistency, usefulness.
  3. Chat Efficiency — conciseness, turn count to resolution.
  4. Client Satisfaction — user-perceived helpfulness.
  5. Compliance — tone, politeness, professionalism, policy adherence.

Per-criterion scores aggregate into a session-level holistic score. Instacart's experiments found binary beat graded 1–10 scales on alignment with human judgment + prompt-engineering cost — canonical wiki instance of concepts/binary-vs-graded-llm-scoring.

Every criterion emits both the binary score and a free-form rationale; rationale is retained for refinement.

2. Three evaluation engines — debate wins

LACE benchmarked three LLM-as-judge engines:

Engine Shape LACE use
Direct Prompting single-pass LLM scoring against criteria baseline
Agentic via Reflection initial score → self-reflection pass that refines; citing Jang 2023 + Madaan et al. alternative
Agentic via Debate Customer-Agent (critical) + Support-Agent (defends), parallel with no inter-agent access → Judge-Agent synthesises; citing Du et al. 2023 production default for context-dependent + subjective criteria

Debate reaches "near-perfect accuracy" on simple compliance criteria and >90% accuracy on context-dependent criteria. See patterns/multi-agent-debate-evaluation and patterns/self-reflection-llm-evaluation.

3. Decouple reasoning from structured output

Each evaluation runs in two passes:

  1. A strong reasoning model (at the time of writing: o1-preview) emits a free-form rationale per criterion.
  2. A separate step — "a separate model or a simple rule-based parser" — converts rationale + verdict into structured JSON (per-criterion True/False + explanation).

Rationale for the split: "producing JSON formatting... can negatively affect performance due to restricted decoding." The decouple lets LACE pick the best reasoning model without being constrained by its JSON-emission quality. Canonical wiki instance of concepts/decouple-reasoning-from-structured-output; a complementary framing to concepts/structured-output-reliability.

Criteria complexity tiers

LACE groups criteria into three tiers by how much context the judge needs:

  • Simple criteria (e.g. professionalism under Compliance). Universal standards; "near-perfect accuracy" with debate.
  • Context-dependent criteria (e.g. contextual relevancy under Answer Correctness). Require Instacart-specific operational knowledge. Canonical named example: a user mentions "my card was declined. My payment method is [a digital wallet]" — the chatbot must know Instacart shoppers use company-authorized cards, not the customer's payment method. Instacart currently embeds this knowledge in a static template in the judge prompt; future work is dynamic RAG-style retrieval to preserve signal-to-noise ratio as the knowledge base grows. Debate: >90% accuracy.
  • Subjective criteria (e.g. answer conciseness under Chat Efficiency). "Low-ROI to refine" — intentionally kept only as a directional regression check; effort spent on improving the chatbot, not the judge, on these axes.

Human-LACE alignment loop

Bootstrap + regression-test the whole framework by:

  1. Human evaluators rate a curated chat set on the same rubric.
  2. LACE rates the same set.
  3. Misalignments drive either:
  4. Criterion-prompt refinement — improved definitions + prompt text ("primary mechanism, applied frequently"), or
  5. Rubric redesign — replacing dimensions with better ones ("used sparingly").
  6. Loop repeats until alignment is strong; re-run on every LACE update to catch regressions.

This is the judge-calibration variant of patterns/human-in-the-loop-quality-sampling and shares structural DNA with Dropbox Dash's patterns/human-calibrated-llm-labeling — humans calibrate the judge; the judge labels at scale. Canonical patterns/human-aligned-criteria-refinement-loop.

Production deployment

  • Sampling: stratified by topic distribution — guarantees coverage of rare but high-impact issue types the chatbot handles infrequently.
  • Surfaces:
  • Trend dashboards — monitor per-dimension quality over time.
  • Drill-down — inspect specific chat sessions flagged for issues.
  • Experimentation-platform integration — LACE verdicts feed directly into chatbot A/B tests so experimental variants are scored against the same rubric at scale, without humans in the per-variant review path.
  • Prompt authoring: all LACE prompts are authored in Markdown with organised sections following the CO-STAR framework; prompt formatting is treated as a first-order quality lever.

Outcomes / numbers

  • Simple criteria (compliance / professionalism): "near- perfect accuracy" with debate.
  • Context-dependent criteria (contextual relevancy): >90% accuracy with debate + static Instacart-knowledge template.
  • Subjective criteria (conciseness): intentionally not pursued past directional-check quality.
  • No per-run latency, cost, or throughput numbers published; no head-to-head win-rate matrix between direct / reflection / debate published.

Cross-references to sibling systems

  • systems/lyft-ai-localization-pipeline — Lyft's LLM-as-judge pipeline for UI-localisation quality; parallel-play architecture (judge → dashboard → experimentation loop), different domain (translation vs. chatbot).
  • systems/zalando-search-quality-framework — Zalando's AI-as-judge for search relevance; same LLM-as-judge → dashboard → experimentation-loop shape, applied to graded (query, product) relevance scoring.
  • systems/instacart-pixel — PIXEL's VLM-based iterative image quality evaluation is a VLM-as-judge sibling at Instacart; shares architectural DNA (evaluator-in-the-loop, iterative refinement, Shishir Kumar Prasad as contributor).
  • systems/instacart-parse — PARSE's entailment-prompt self-verification + LLM-as-judge quality screening is the structured-extraction sibling.

Source

Last updated · 517 distilled / 1,221 read