PATTERN Cited by 1 source

Context compaction for serving cost

Pattern

Systematically reduce the token count of a verbalized prompt through signal-aware compression — retaining high-value information, omitting low-signal events, summarizing repetitive patterns, and structuring for prefix caching — to achieve proportional serving cost reduction with negligible quality degradation.

Problem

In LLM-backed systems where serving cost is proportional to context length (especially in prefill-only mode), naively verbalizing every user interaction quickly exceeds the token budget. The cost of longer prompts is both compute (more FLOPs) and memory (larger KV cache).

Solution

Three-step optimization:

1. Clean and compress events: Drop low-signal engagements (short plays, hovers), compress repetitive behavior (binge-watching sessions).
2. Find the elbow point: Vary included history length and plot quality metric (MRR) vs. event count. Identify the diminishing-returns threshold.
3. Optimize verbosity: For retained events, test different detail levels and simplified wordings, measuring quality at each step.

Additionally, structure prompts for shared-prefix caching — maximize the common prefix across requests so the inference engine can reuse cached KV states.

Consequences

• Netflix achieved ~1/3 token budget with negligible offline metric degradation.
• Serving cost reduction is approximately proportional to context length reduction.
• Forces explicit information-theoretic reasoning about which signals matter.
• Shifts modeling effort from feature engineering to context engineering.
• Compound benefit with prefix caching — structured prompts further reduce compute.

Seen in

• sources/2026-07-30-netflix-genrec-towards-llm-native-recommendation — "we can reduce the context tokens to roughly one-third of the original budget with negligible degradation in offline ranking metrics. Since serving cost is approximately proportional to context length, we observed a similar reduction in serving cost."