Skip to content

CONCEPT Cited by 1 source

LLM hyperlink hallucination

Definition

LLM hyperlink hallucination is the failure mode where a language model fabricates URLs in its output — emitting plausible-looking, well-formed hyperlinks that do not exist in the source corpus and do not resolve to real pages.

Sub-class of concepts/llm-hallucination specific to URL emission. Sibling on the wiki to concepts/llm-icon-hallucination (Vercel v0, 2026-01-08; LLM references library symbols / icon names that don't exist in churning library namespaces).

Canonical wiki disclosure: Yelp's 2026-05-27 LLM-Assisted CS Chatbot post.

"One of the most notable unexpected challenges was the tendency of Large Language Models (LLMs) to hallucinate hyperlinks frequently. Since our knowledge base articles contain numerous hyperlinks, and we intended for the LLM-generated responses to include accurate links, this required a dedicated solution." (Source: sources/2026-05-27-yelp-beyond-the-menu-tree-how-yelp-built-a-smarter-customer-success-chatbot)

Hyperlinks are a uniquely high-confidence-but-fragile output class:

  • Plausible structure. URLs follow a strict surface format (https://domain/path/segment); models trained on web text can generate URL-shaped strings that look right.
  • Cited as authoritative. A user reading the LLM's response treats hyperlinks as first-class evidence — clickable, verifiable. A fabricated link breaks user trust when it 404s.
  • Combinatorial space. The set of valid URLs is sparse within the URL-shaped string space. Even small token-level errors (/help/refunds → /help/refund) produce non-resolving URLs.
  • Training-data leakage. The model has seen many real URLs during pretraining; it generates from a learned distribution over URL paths that may not match the user's specific knowledge base. A model trained on the open web may emit "plausible Yelp help URLs" that don't correspond to any real Yelp Support Center article.
  • In-context-prompt URLs are not enough. Even when the retrieved-context articles contain the correct URLs, the model may mix and match — taking a path fragment from one article and a query parameter from another. The "present-but-misassembled" URL is the canonical fabrication shape.

Mitigation — allowlist validation

The structural fix is deterministic post-hoc validation, not prompt engineering:

  1. Extract URLs from the retrieved-context articles (the articles that grounded the LLM's response). This forms the per-response allowlist.
  2. Parse the LLM output for URL-shaped tokens.
  3. Validate each URL against the allowlist (exact match, not fuzzy).
  4. Strip / reject / replace any URL not on the allowlist.

The Yelp post discloses the structural mitigation (allowlist extraction + validation pass) without revealing the specific match algorithm or replacement policy. Verbatim:

"To counteract this, we developed a process to reliably retrieve valid hyperlinks from the source articles and integrated specific validation checks. This verification process ensures that any link included in the final response genuinely originates from one of the retrieved Support Center articles and is not invented by the LLM." (Source: sources/2026-05-27-yelp-beyond-the-menu-tree-how-yelp-built-a-smarter-customer-success-chatbot)

(See patterns/hyperlink-allowlist-validation-on-llm-output for the full pattern shape.)

Why prompt-engineering alone doesn't work

Common naive mitigations and their failure modes:

  • "Only use links from the provided context" prompt — model ignores the instruction or partially obeys (path-from-A, query-from-B mix-and-match).
  • Few-shot URL examples in prompt — model overfits to the example URL structure, fabricates novel URLs that match the pattern.
  • Stop-tokens around URLs — model emits URLs in non-stopped positions or breaks the markdown link syntax.

The structural property that breaks prompt-engineering: URL correctness is binary (the URL resolves or it doesn't), unlike factual claims where graded faithfulness is achievable through better prompting. Binary-correctness output classes require deterministic post-hoc validation.

  • vs concepts/llm-icon-hallucination (Vercel v0, 2026-01-08) — both are binary-correctness symbolic-space hallucinations. Vercel v0's icon hallucination (LLM references React component names from churning library namespaces that don't exist) is mitigated by embedding-based name resolution inside streaming output rewrite — same structural philosophy: deterministic post-hoc fix-up rather than prompt engineering. Hyperlink hallucination uses allowlist exact match rather than embedding resolution because URLs don't have a useful semantic similarity space — /help/refund ≠ /help/refunds is a binary mismatch, not a graded one.
  • vs general factual hallucination — factual hallucinations have graded faithfulness mitigations (factuality decoding, credibility scoring, narrative-coherence filtering); hyperlink hallucinations require binary post-hoc validation.
  • vs concepts/web-search-telephone-game — also a URL- related failure mode (RAG-via-web-search with summariser telephone game) but the failure is content distortion, not URL fabrication.

Caveats

  • Single-source canonical on the wiki. Yelp's 2026-05-27 post is the wiki's first explicit canonicalisation. The failure mode is broadly known in industry but rarely documented at this altitude.
  • Allowlist scope. Whether the allowlist includes URLs that appear in the article body (and not just article-page URLs themselves) is the practical implementation question. Yelp's verbatim "valid hyperlinks from the source articles" implies article-body URLs are extracted, but the precise extraction logic isn't disclosed.
  • External-link policy. Whether arbitrary external URLs (e.g. linking to another product's website) are blocked by the allowlist or pass through is unclear.
  • No quantitative rate disclosed. Yelp doesn't quantify how often hyperlink hallucinations were occurring before the validation gate was added; the "frequently" qualitative framing is the only signal.

Seen in

Last updated · 542 distilled / 1,571 read