Skip to content

CONCEPT Cited by 1 source

Lost in the middle effect

Definition

The "lost in the middle" effect is a named failure mode of long-context LLM prompting where "details in the middle of long inputs are often overlooked or distorted" — the model attends strongly to content near the beginning and end of its context window but degrades sharply on middle-of-context information, independent of whether that information is actually relevant (Source: sources/2025-09-24-zalando-dead-ends-or-data-goldmines-ai-powered-postmortem-analysis).

Canonical framing per Zalando's 2025-09-24 postmortem-analysis- pipeline post, which names it explicitly as the reason they chose a multi-stage pipeline architecture over a single large-context prompt:

"While large context windows allow models to process more information, we observed a 'lost in the middle' effect, where details in the middle of long inputs are often overlooked or distorted. In addition, large contexts do not guarantee perfect recall and can increase latency, memory usage, and cost."

Why it matters for system design

The failure mode pushes back on the naive frame "larger context is strictly better." Three practical consequences:

  • Recall is not uniform over position. A prompt containing N documents, where the critical fact sits at positions 30–70% through the sequence, can produce a model output that behaves as if that fact weren't present — plausible, confident, wrong. This interacts with hallucination: the model confabulates to fill the gap rather than reporting the actual missed datum.
  • Latency + memory + cost scale with context length. Even when the model's recall holds, putting N documents in one prompt multiplies inference cost by ~N and often increases per-token latency non-linearly. Zalando's original NotebookLM iteration "requires about 5 minutes to read the summary and make a conclusion about root causes" — but over thousands of documents "sifting through summaries takes weeks for a dedicated team of experts."
  • Opacity of failures. When a multi-document prompt drops information, the model doesn't surface "I couldn't find X in the middle of your input" — it produces a well-structured output as if X weren't supplied. You only detect the drop by sampling against ground truth.

The multi-stage alternative

Zalando's fix is canonicalised as patterns/multi-stage-llm-pipeline-over-large-context: decompose the task so each stage's input size is bounded (typically one document or a small set of digests), each stage has a single objective, and each stage's output is human-readable. The map-fold functional shape is the composition primitive — map over documents (per-document extraction), fold into a higher-level synthesis.

This trades raw context capacity for per-stage reliability:

  • Each stage runs on a bounded input, dodging the lost-in-the-middle regime.
  • Each stage's output is inspectable — a human can curate before the next stage consumes it.
  • Each stage's latency is bounded and parallelisable.

The explicit trade-off stated in the Zalando post:

"we designed a multi-stage LLM pipeline instead of using high-end LLMs with large context windows. It is a deliberate design trade-off aimed at simplicity and reliability."

  • LLM hallucination — the model fills missed middle-of-context information with confabulation. Lost-in-the-middle is a specific mechanism that triggers hallucination in long prompts.
  • Over-packing. Even without lost-in-the-middle, dumping thousands of documents into a prompt is expensive and opaque — you lose the ability to attribute which document contributed to which output claim.

Seen in

  • sources/2025-09-24-zalando-dead-ends-or-data-goldmines-ai-powered-postmortem-analysis — canonical wiki instance. Zalando's datastore SRE team explicitly cites lost-in-the-middle as the motivation for chaining five narrow-objective LLM stages instead of a single large-context prompt. Observed empirically via NotebookLM over thousands of postmortems — "severe hallucinations and loss of the incident context by LLM while producing summaries."
Last updated · 507 distilled / 1,218 read