Skip to content

CONCEPT Cited by 1 source

Narrative coherence as hallucination filter

Definition

Narrative-coherence-as-hallucination-filter is the architectural technique of running a second pruning pass over already-credibility-scored findings in which each finding is evaluated not on its own evidence but on how well it fits the emerging story. Findings that contradict or lack support from the broader narrative are pruned, even if they individually scored above the per-finding plausibility threshold.

The load-bearing canonical claim (Source: sources/2026-04-13-slack-managing-context-in-long-run-agentic-applications):

"A hallucination can only survive this process if it is more coherent with the body of evidence than any real observation it competes with."

This is a two-stage hallucination filter: point-wise credibility scoring (the first stage; see concepts/credibility-scoring-rubric) gates at the individual-finding level; narrative coherence (this concept) gates at the story level. A hallucination would have to survive both to reach the final report.

Named and canonicalised by Slack's Security Engineering team for the Spear Critic's Timeline task.

Why a second stage

The premise is that a sophisticated hallucination can:

  • Look plausible in isolation (pass credibility scoring at a moderate-to-high band).
  • Cite real-looking tool calls (the Critic has introspection tools but not infallible data-reading).
  • Be internally consistent with its own justification.

What it cannot easily do is fit coherently into the broader chronology of the investigation. If the investigation shows user X logged in at 09:30, and a hallucinated finding claims user X was offline at 09:32, the hallucinated finding breaks the temporal narrative regardless of how plausible it looked in isolation.

The Timeline task operationalises this: it forces every surviving finding to fit into a consolidated chronological narrative built from multiple Experts' credible findings. Findings that don't fit get pruned.

Mechanism (Slack's Timeline task)

The Critic's Timeline task runs after the Critic's Review task (where per-finding credibility scoring happens) and before the Director's next decision. Its inputs are three existing artifacts:

  1. The most recent Review — annotated findings with credibility scores.
  2. The previous Critic's Timeline — the running chronology from prior rounds.
  3. The Director's Journal — the Director's planning state.

Its output is a consolidated timeline + a narrative-coherence score + top-3 gaps.

Four explicit consolidation rules (Source: sources/2026-04-13-slack-managing-context-in-long-run-agentic-applications):

  1. Include only events supported by credible citations — speculation doesn't belong on the Timeline.
  2. Remove duplicate entries describing the same event — an event shouldn't appear twice because two Experts mentioned it.
  3. When timestamps conflict, prefer sources with stronger evidence — a log entry timestamp beats an inferred time.
  4. Maintain chronological ordering based on best available evidence — events must flow logically in time.

Narrative-coherence rubric (distinct from credibility)

Slack discloses a 5-level rubric specifically for the Timeline's coherence score (Source: sources/2026-04-13-slack-managing-context-in-long-run-agentic-applications):

Score Label Meaning
0.9-1.0 Trustworthy Strong corroboration across multiple sources, consistent timestamps, no significant gaps
0.7-0.89 Highly-plausible Good evidence support, minor gaps present, mostly consistent Timeline
0.5-0.69 Plausible Some uncertainty in event ordering, notable gaps exist
0.3-0.49 Speculative Poor evidence support, significant gaps, conflicted narrative
0.0-0.29 Invalid No evidence, confounding inconsistencies present

Same numeric bands as the per-finding rubric, but rebased from credibility-per-claim to coherence-of-the-whole-story. Two rubrics, two stages, two distinct hallucination filters.

Why this works (the coherence argument)

The argument is a structural one about hallucination behaviour:

  • Truthful observations are consistent with reality, so they naturally fit together into a coherent chronology.
  • Hallucinations are uncorrelated fabrications, so the probability that multiple independent hallucinations happen to fit a shared narrative is low.

Consequently, forcing findings to fit a narrative acts as a dilution filter for hallucinations: each hallucination has to compete for narrative slot with real observations, and real observations have the structural advantage of being consistent with each other.

Slack's verbatim summary:

"The Timeline task raises the bar for hallucinated findings by enforcing narrative coherence. To be preserved, each finding must be consistent with the full chain of evidence; findings that contradict or lack support from the broader narrative are pruned. A hallucination can only survive this process if it is more coherent with the body of evidence than any real observation it competes with."

Operational property: task separation keeps it honest

Slack explicitly separates the Timeline task from the Review task. The Review is token-intensive and tool-call-heavy (methodology audit). The Timeline "operates entirely on data in the prompt" — no tool calls, no methodology audit.

The intuition (Source: sources/2026-04-13-slack-managing-context-in-long-run-agentic-applications):

"The more narrowly scoped task leaves a greater capacity for reasoning in the problem domain, rather than methods of data gathering or judgements of Expert methodology."

A single combined Review+Timeline task would ask the Critic to simultaneously audit methodology and assemble coherent narrative — two different cognitive loads. Task separation is the canonical patterns/one-model-invocation-per-task move applied to the Critic role.

The "most plausible" framing

A subtle point: the Timeline is "the most plausible consolidated timeline," not "the true timeline" or "the definitive timeline." The coherence-filter argument does not claim certainty; it claims best available reconstruction given the evidence + consistency constraints.

Gaps are a first-class part of the Timeline's output — three types explicitly identified (evidential / temporal / logical), capped at top 3 (see concepts/gap-identification-top-n). The Timeline is honest about its own incompleteness.

Contrasts

  • vs. concepts/credibility-scoring-rubric — credibility is point-wise (per finding); narrative coherence is global (across findings). Both are 5-level rubrics, but they grade different properties.
  • vs. classical fact-checking — fact-checking compares claims against external ground truth. Narrative coherence is internal consistency: it doesn't require ground truth, only mutual consistency of the evidence.
  • vs. LLM-as-judge — LLM-as-judge evaluates individual outputs. Narrative coherence evaluates compositions of outputs against each other.
  • vs. ensemble voting — ensemble voting cross-checks multiple model outputs on the same question. Narrative coherence cross-checks multiple findings on different questions against a shared story.

Seen in

Last updated · 470 distilled / 1,213 read