Skip to content

Advancing Generative Recommendation Inference: Spotify × NVIDIA on TensorRT-LLM

Summary

Spotify is moving its recommendation stack from traditional multi-step retrieve-then-rank systems toward a Large Personalized Language Model that serves recommendations across surfaces (Home, next-song, "talk to Spotify") in real time. The engineering challenge is the inference stack: three distinct inference techniques Spotify needs — efficient beam search for candidate generation, candidate scoring via model log-probabilities, and soft-token personalization from existing user embeddings — were each best supported by a different open-source engine (vLLM, SGLang, and nothing good for soft tokens respectively). Spotify partnered with NVIDIA to unify all three into a single engine, TensorRT-LLM (TRT-LLM), closing its feature gaps so one engine could serve the whole workload at high GPU utilization. The result: on the Qwen3 1.7B variant, up to 54% cost reduction for beam-search workloads and up to 7% for scoring workloads, driven by TRT-LLM implementing beam search inside the sampler (lower host overhead + overlap scheduling) rather than vLLM's outside-the-engine approach.

Key takeaways

  1. Generative recommendation = one LLM replacing the retrieve/rank pipeline. Spotify's bet is a single Large Personalized Language Model serving multiple surfaces in real time, replacing per-surface recommendation systems with their separate candidate-retrieval and ranking stages (retrieval→ranking funnel). First shipped for podcast recommendations in January 2026, grounded in the PLUM paper (arXiv:2510.07784) and Spotify's own semantic-ID generative-retrieval work (arXiv:2603.17540 — "Deploying Semantic ID-based Generative Retrieval for Large-Scale Podcast Discovery"). (Source: this article)

  2. Beam search is the candidate-generation primitive, and engine support varies wildly. Spotify uses beam search to explore the LLM's output space, keeping the most promising token sequences active at each decode step; the top sequences are semantic IDs translated back to catalog items. The initial January launch ran on vLLM, but "vLLM's beam search implementation operates outside the core GPU engine, which introduces CPU-side bottlenecks and additional GPU work due to the recomputation of partially filled KV blocks" — so Spotify could not saturate high-end GPUs and deliberately chose lower-tier accelerators to keep the system at an optimal load point and manage cost. (Source: this article)

  3. Candidate scoring uses log-probs, and SGLang's masking trick scores many candidates in one request. Candidate scoring ranks a candidate set by the model's log-probabilities. The naive approach is a fan-out — one inference request per candidate, extract logprobs, aggregate — which is available everywhere but wasteful. SGLang has a more efficient candidate-masking technique (its "Score API") that scores multiple candidates in a single request; the feature was contributed partly by LinkedIn's engineering team. Spotify adopted SGLang specifically to get this. (Source: this article)

  4. Three personalization approaches; soft tokens won. Spotify explored three ways to inject user preference into the LLM prompt:

  5. Plain user text — serialize listening/interaction history into prompt text. Accurate representation needs huge prompts → high latency (over Spotify's request budget) and high GPU use. Used sparingly (e.g. the user's most recent real-time streams).
  6. User semantic IDs — extend the model vocabulary with user-semantic-ID tokens (à la PLUM); a user is a handful of text tokens, keeping prompts short with no custom inference needed. Promising but not yet production-ready.

  7. Soft tokens (the winner) — learn a projection from the existing user-embedding space into the LLM's embedding space as a single soft token (the "Embedding to Prefix" approach, arXiv:2505.17051, Spotify Research). The LLM is fine-tuned on a subset of weights to consume this soft token; at inference a user vector is projected and inserted into the prompt before the first forward pass. Already used for podcast recs; being integrated for other tasks on the internal LLM. (Source: this article)

  8. The unification problem: no single engine had all three features. Beam search was best on TRT-LLM (superior hardware utilization); candidate scoring was best on SGLang (Score API); soft tokens + multi-beam sequence return were on vLLM. Running three engines is operationally untenable as the workload grows, so Spotify's goal with NVIDIA was explicitly: "unify the inference features that we require (efficient beam search, scoring, and user soft tokens) into a single engine." (Source: this article)

  9. What the Spotify × NVIDIA collaboration landed upstream in TRT-LLM. Working with NVIDIA engineers (Marek, Martin, Stefan, Alexander), the gaps were closed with concrete upstream releases (upstream + in-house in parallel):

  10. returning multiple sequences in beam search (v1.2.0rc3)
  11. out-of-tree model support in trtllm-serve (v1.2.0rc4)
  12. passing multi-modal embeddings via image_embeds (v1.3.0rc0)
  13. multi-item scoring via the Python API (v1.3.0rc20)

  14. general bug fixes / perf improvements. (Source: this article)

  15. Beam search in the sampler is why TRT-LLM wins. "Beam search in TensorRT-LLM is implemented as part of the sampler, which reduces host overheads and, combined with overlap scheduling, ensures good GPU utilization." The C++ sampler was also replaced with a more flexible PyTorch-heavy design, and the new beam-search path gave a "notable performance uplift" over the previous implementation. This is the architectural root of the cost win vs vLLM's outside-the-engine beam search. (Source: this article)

  16. Soft tokens were implemented as a TRT-LLM out-of-tree Qwen3 plugin. Using TRT-LLM's out-of-tree model support, Spotify built a customized Qwen3 extension plugin that passes user vectors via multi-modal request parameters and performs the user-embedding projection + soft-token replacement inside the engine. (Source: this article)

  17. Multi-item scoring reaches SGLang parity but still needs a custom serving layer. TRT-LLM's multi-item scoring Python API gives functionality equivalent to SGLang's scoring API; Spotify still needs "a custom serving layer implementing efficient tokenization and request batching." The value is explicit: multi-item scoring "avoids the overhead of a single request per candidate, while retaining a computational effort linear in the number of candidates" — the economic argument against the fan-out approach (batch over per-request overhead). TRT-LLM also recently added serving of prefill-only embeddings (PR #15424). (Source: this article)

Operational numbers

  • Qwen3 1.7B (the model variant benchmarked): up to 54% cost reduction for beam-search workloads, up to 7% cost reduction for scoring workloads from the TRT-LLM implementation vs the prior baseline. (Source: this article)
  • First LLM-powered recommendation launch: podcast recommendations, January 2026, on vLLM with lower-tier accelerators chosen for cost/load reasons.
  • TRT-LLM upstream releases carrying the Spotify-driven features: v1.2.0rc3, v1.2.0rc4, v1.3.0rc0, v1.3.0rc20.

Systems / concepts / patterns extracted

Caveats

  • The 54% / 7% figures are for a single model size (Qwen3 1.7B) against an unspecified baseline (presumably the prior vLLM/SGLang path); no percentile, absolute $/req, QPS, or GPU-type is disclosed. Treat as directional, not a portable benchmark.
  • Beam-search comparison and multi-item-scoring charts are referenced as images only; the raw numbers behind the charts are not in the text.
  • User semantic IDs are explicitly not production-ready; soft tokens are the only personalization path confirmed in production (podcasts), being integrated elsewhere.
  • Multi-item scoring on TRT-LLM still requires a Spotify-built custom serving layer (tokenization + batching) — parity with SGLang is at the API level, not turnkey.
  • The road-ahead items (quantization, Blackwell-specific techniques) are future work, not shipped.

Source

Last updated · 854 distilled / 2,448 read