Yelp Menu Vision¶

SYSTEM Cited by 1 source

Menu Vision is a Yelp mobile feature (iOS + Android, launched end of October 2025) that lets a diner point a phone camera at a restaurant menu and see, overlaid in an augmented-reality view, which dishes Yelp has photos and reviews for. Behind the simple interaction is a system that combines on-device text recognition, on-device fuzzy matching, a precomputed per-business dish store in Cassandra, and a server-side feature-eligibility gate. It began as a two-day hackathon prototype and was productionized in ~six weeks. (Source: sources/2026-08-20-yelp-building-menu-vision-real-time-dish-recognition)

Architecture¶

1. Data source (offline). Menu Vision does not build a new corpus; it reuses and widens existing Yelp menu data. It combines owner/partner-provided menus with in-house curated dishes inferred from reviews and — new for this feature — user-submitted photo captions. The offline path is:

Combine all menu sources.
Deduplicate so each dish appears once.
Filter to only dishes with ≥1 associated photo (the feature only needs dishes it can show a photo for).
Store the final per-business collection in Cassandra for fast, reliable retrieval.

Pre-processing offline and serving from Cassandra is what let Yelp expand coverage "without building a new system from scratch" while consistently meeting its low-latency targets. See dedup-filter-precompute-to-fast-store.

2. On-device recognition + matching (client). Text recognition, text cleanup, and dish matching all run on the client using each platform's native ML frameworks — no images are sent to the server, which minimizes latency and removes dependence on connection speed. See concepts/on-device-ml-inference and concepts/on-device-ml-inference. When a business page opens, Menu Vision prefetches and locally stores the business's dish collection, so launching the scanner needs no further client-server round trips (prefetch-on-page-open). On iOS, Apple VisionKit supplies real-time scanning guidance ("Slow down", adjust angle) to improve the input signal, and QR-code detection for digital menus.

3. Feature surfacing (server). The contextual "Scan the Menu" prompt at the business entry point is gated server-side, powered by Yelp's existing educator framework (impression tracking, frequency capping, experimentation). Per business-page view the backend evaluates:

Location verification — GPS distance between user and restaurant; shown only when the user is physically near the restaurant.
Content availability — datastore query confirming the business meets a minimal dish-photo threshold.
Frequency capping — prompt-frequency limits + cooldown after dismissal.

Keeping this logic server-side keeps the client thin and lets targeting rules change without an app release. Two permanent entry points (photo-search toolbar button; business-page menu-section button) complement the contextual prompt. See server-side-feature-eligibility-gate.

The matching algorithm¶

The initial launch used exact character-for-character matching and was too strict — "Garlic Noodle" (singular) or "Garlic Noodles w/ Pork" failed against "Garlic Noodles". The enhanced matcher is three-phase (see tiered-exact-substring-fuzzy-matching and multi-phase-fuzzy-matching):

Exact match against the primary dish name and all synonyms.
Bidirectional substring match — recognized-text-contains-dish OR dish-contains-recognized-text (handles modifiers, e.g. "Spicy Pad Thai (V)" → "Pad Thai").
Jaro-Winkler similarity fallback — prefix-weighted, so it favors matches where the leading tokens agree (dish names lead with the key identifier) and tolerates spelling variants, OCR errors, and transliterations.

Data-quality enhancement (LLM pipeline)¶

To fix coverage and naming-inconsistency gaps ("Pad Thai" in one dataset vs "Thai-Style Stir Fried Noodles" in another), Yelp added a three-step LLM pipeline that produces one unified, standardized menu per business: standardize partner menus (normalize names, extract synonyms, tag price / portion / dietary / calories) → process customer language from reviews and photo captions → intelligently combine and deduplicate with popularity indicators. See llm-menu-normalization-pipeline.

Fallbacks¶

Inventory fallback — if the scanner detects no dishes, it shows the restaurant's full dish inventory (popular first), so the feature is still useful when recognition struggles (inventory-fallback-on-empty-recognition).
QR-code detection (iOS) — point at a menu QR code → prompt to open the restaurant's digital menu.

Results¶

Launched end of October 2025 on both platforms with a gradual staged rollout, monitoring client-side logs, server-side load, and error rates; no major bugs or crashes reported. Menu Vision increased retention for users who used it. An April 2026 update replaced text pills with rich, photo-driven dish cards (image, name, photo/review counts, "Popular" badge, price) in a swipeable paginated carousel over the live camera view.

Seen in¶

sources/2026-08-20-yelp-building-menu-vision-real-time-dish-recognition — the canonical (and only) source; full architecture, matching algorithm, and LLM normalization pipeline.