Skip to content

ZALANDO 2026-10-11 Tier 2

Read original ↗

Automating Data Ingestion with a RAG Flywheel

Summary

Zalando's Supply Operations team automates schema mapping — translating each supplier's idiosyncratic spreadsheet headers (10–60 columns per sheet, with names like Art.-Nr., Article Number, SKU, UVP, Farbe) into a strict internal target schema of 55+ columns — using an LLM-powered continuous-learning loop they call a "RAG Flywheel." Legacy approaches (hardcoded rules, regexes, Levenshtein fuzzy matching) break on real-world header variability because they lack semantic understanding and need constant engineer maintenance per new supplier. The new system splits work across AWS Bedrock: Amazon Titan Text Embeddings V2 generates 1024-dim vectors for semantic search, and a generative model (Anthropic Claude Sonnet or Gemini) does the mapping reasoning. The architecture's reliability comes from bounded responsibilities: the model proposes a mapping, tool use / function calling constrains the response shape (structured output), and the API validates every mapping value against the known source and target columns before anything reaches users or the database — the LLM output is treated as untrusted until verified. Mappings graduate from a cold-start zero-shot prediction to a warm-start RAG prediction: once a human approves a mapping, it is asynchronously embedded and stored in pgvector (PostgreSQL + HNSW index), so future uploads retrieve historically-approved mappings via cosine-similarity search (≥0.8 threshold, top-3 per column, ranked by approval count to favor consensus over anomaly) and inject them as <historical_context> into the prompt. Every suggestion stays human-reviewed; the approved decision is the training data that improves the next prediction. Reported acceptance rate: 85%, rising as the historical index grows. The pgvector history also provides graceful degradation — when the LLM is unavailable, the system can still return a previously-approved mapping from retrieval alone.

Key takeaways

  1. The "RAG Flywheel" is a human-feedback continuous-learning loop, not a training loop. A cold-start zero-shot prediction becomes useful historical knowledge only after a human reviews and approves it; future uploads retrieve those approved mappings, rank them by similarity and past usage, and present better suggestions. "Human feedback is therefore not just a correction step; it is the data that improves the next prediction." The model's weights never change — the retrieval corpus grows. (Source: sources/2026-10-11-zalando-automating-data-ingestion-with-a-rag-flywheel)
  2. Bounded responsibilities make an unreliable component reliable. The design splits the problem into three bounded parts: the model proposes, tool use constrains the response shape, the API verifies every result is valid before it reaches users or downstream systems. "Tool use constrains the response shape, but it does not guarantee that the semantic mapping is correct; that is why every suggestion is validated and reviewed by a human." Shape-validity and value-validity are separate guarantees enforced at separate layers. (Source: sources/2026-10-11-zalando-automating-data-ingestion-with-a-rag-flywheel)
  3. The API is the gatekeeper; LLM output is untrusted until verified. Before any suggestion reaches users or the DB, synchronous validation rejects unknown source columns (not in the uploaded file), unknown target columns (not in the internal schema), and enforces full coverage of all source columns, plus bounds-checks the confidenceScore (0–100) and reorders suggestions to match the original file structure. A single invalid mapping throws and fails the whole batch — fail-closed. (Source: sources/2026-10-11-zalando-automating-data-ingestion-with-a-rag-flywheel)
  4. Two API-level constraints bound LLM behavior: zero temperature + forced tool use. Temperature is hardcoded to 0.0 because "schema matching benefits from repeatable reference lookups," and tool_choice forces the model to call submit_column_mappings, which returns a fixed {sourceColumn, targetColumn, reasoning, confidenceScore} structure. The system prompt injects the target schema (<target_schema>) and enforces a null fallback rule — "If a source column does not logically match ANY of the target columns, you MUST return JSON null ... Do not force a bad match." (Source: sources/2026-10-11-zalando-automating-data-ingestion-with-a-rag-flywheel)
  5. Semantic embeddings beat string matching for header equivalence. UVP and rrp_de are semantically equivalent but string-matching fails; so each source column is embedded as a concatenated string `${col.name} | ${col.dataType} | samples: ${col.samples.slice(0,5)}` — name + inferred type + representative sample values — and sent to Titan Text Embeddings V2 to produce a 1024-dim vector that captures the column's semantic meaning. (Source: sources/2026-10-11-zalando-automating-data-ingestion-with-a-rag-flywheel)
  6. The warm-start retrieval query ranks by consensus, not just similarity. The pgvector SQL uses the cosine-distance operator <=> (similarity = 1 - cosine_distance), filters to ≥0.8 cosine similarity, groups by (source_column, target_field), and orders by approval_count DESC, similarity DESC — "a mapping confirmed multiple times across different uploads naturally outweighs a single anomaly" — then caps at top-3 per column to keep the prompt focused and "avoid context pollution." The LIMIT 3 is per incoming column, not per file. The warm-start prompt tells the model to prioritize historical mappings (>90 confidence for `approvalCount

    1`, >80 for single-upload matches), falling back to logical deduction otherwise. (Source: sources/2026-10-11-zalando-automating-data-ingestion-with-a-rag-flywheel)

  7. The retrieval history provides graceful degradation when the LLM is down. "The system can continue to provide useful suggestions when the LLM is temporarily unavailable" — for a new column it queries pgvector and returns a previously-approved mapping when similarity and approval thresholds are met, "instead of failing completely." The fallback still routes through human review, and columns without a sufficiently similar historical match stay unresolved rather than being mapped by force. Fallback quality depends on the availability of representative approved mappings. (Source: sources/2026-10-11-zalando-automating-data-ingestion-with-a-rag-flywheel)
  8. The historical index is deliberately rebuildable. Embeddings for old mappings can be deleted from the DB; once removed, "they are no longer considered by future historical checks." This gives a simple lever to reset stale or incorrect history "without changing the rest of the mapping workflow" — keeping the learning loop under engineering control. (Source: sources/2026-10-11-zalando-automating-data-ingestion-with-a-rag-flywheel)
  9. Supplier data is untrusted input and must be segregated from instructions. Supplier column names and sample values "should be treated as untrusted data, kept clearly separated from system instructions ... a value that looks like an instruction must remain data rather than becoming part of the prompt's instructions" — the article's explicit prompt-injection defense. The same boundary applies to retrieved historical mappings, and the post notes embeddings are not a substitute for anonymisation — they can still encode information about the source data, so access controls, retention, logging, and deletion must cover both raw supplier data and its embeddings. (Source: sources/2026-10-11-zalando-automating-data-ingestion-with-a-rag-flywheel)
  10. The broader lesson: LLMs are most useful inside a controlled workflow. "Retrieval provides relevant context, validation protects the contract, and human approval keeps the learning loop grounded in operational reality." The model is the proposer, never the final decision boundary. (Source: sources/2026-10-11-zalando-automating-data-ingestion-with-a-rag-flywheel)

Systems / concepts / patterns extracted

  • RAG — the central mechanism; the "flywheel" is the warm-start regime where human-approved mappings become the retrieval corpus that grounds future predictions. Distinct from the document-chunk RAG norm: the retrieved items are (source_column → target_field) mappings with approval counts, not document chunks.
  • Human-in-the-loop — every suggestion is reviewed; the approved decision is the data that grows the retrieval index. Human review gates both the primary path and the LLM-down fallback.
  • Vector embedding — Titan Text Embeddings V2, 1024-dim; the embedding payload concatenates name + inferred type + sample values to capture column semantics.
  • Vector similarity search — cosine similarity via pgvector's <=> operator, ≥0.8 threshold, top-3, ranked by approval count then similarity.
  • Structured-output reliability — forced tool use (tool_choice) constrains the response shape; separately, the API validates the response values. Shape-valid ≠ semantically-correct.
  • Graceful degradation — the pgvector history serves retrieval-only suggestions when the embedding/generative models are unavailable, rather than failing the workflow.
  • Prompt injection — supplier column names/values and retrieved history treated as untrusted data, segregated from system instructions.
  • Cold start — the zero-shot regime for a never-seen schema with "NO HISTORICAL CONTEXT AVAILABLE," relying purely on the target-schema definitions and the model's general knowledge.
  • Entity resolution — schema matching is a column-level entity-resolution problem (which source header is which target field) solved semantically rather than by string distance.
  • LLM hallucination — the explicit threat the gatekeeper API defends against ("LLMs are not deterministic: they can produce different outputs and hallucinate").
  • Flywheel pattern — same close-the-loop-against-human-feedback shape, here on retrieval corpus growth rather than prompt optimization; cross-referenced as a sibling flywheel.
  • API-as-gatekeeper / validate-LLM-output-before-use — recorded as prose + tag only (single source; maps into concepts/structured-output-reliability and patterns/deterministic-tool-vs-llm-judgment); no page minted.
  • Amazon Bedrock — hosts both the embedding model (Titan Text Embeddings V2) and the generative model; the Bedrock Mantle endpoint is OpenAI-SDK-compatible, so Zalando uses the standard OpenAI Node SDK (baseURL: https://bedrock-mantle.<region>.api.aws/v1) for function calling.
  • pgvector · PostgreSQL · HNSW — the vector-storage substrate for the historical mappings.
  • Titan Text Embeddings V2, OpenAI SDK, Bedrock Mantle — named in systems: frontmatter; plain-text mentions, no pages (not primary subjects, not listed elsewhere).

Operational numbers

  • Internal target schema: 55+ columns (e.g. SKU, Color Code, Color Name).
  • Supplier source sheets: 10–60 columns per sheet per workbook.
  • Embedding model: Amazon Titan Text Embeddings V2, 1024-dimensional vectors.
  • Generative model: temperature 0.0 (hardcoded).
  • Cosine-similarity retrieval threshold: ≥0.8 (cosine distance ≤0.2).
  • Retrieval cap: top-3 historical matches per incoming source column.
  • Confidence-scoring guidance: 95–100 exact/identical-name match down to 1–49 low-confidence guess; warm-start prompt asks for >90 when approvalCount > 1, >80 for single-upload matches.
  • Embedding payload: first 5 sample values per column.
  • Reported production acceptance rate: 85% of suggestions accepted without changes, increasing as the historical index grows (self-reported from live workflows, not a labeled-dataset benchmark).

Caveats

  • Not a formal benchmark. The 85% acceptance rate "reflects human judgments in live workflows rather than a formal evaluation against a labeled dataset"; full analytics are planned "later in the year." No precision/recall, no per-supplier breakdown, no error analysis.
  • Single-domain narrative. This is one Supply Operations schema-mapping system; the numbers (1024-dim, 0.8 threshold, top-3, 55+ target columns) are this system's configuration, not a general recipe.
  • Model specifics partly abstracted. The generative model is shown as anthropic.claude-3-sonnet in a "// or any other generative model" example; the post names Claude Sonnet and Gemini interchangeably, so the exact production model/version is not pinned.
  • camelCase vs snake_case boundary. The API uses sourceColumn / confidenceScore; the DB uses source_column / target_field — the post flags these as "the same concepts at different boundaries," not two systems.
  • Fallback quality is conditional. The LLM-down retrieval fallback is only as good as the available approved mappings, the embedding model, and the chosen similarity threshold; unmatched columns stay unresolved rather than force-mapped.
  • Code is "simplified." The TypeScript validation function and prompts are presented as simplified illustrations, not the production source.

Source

Last updated · 854 distilled / 2,448 read