PATTERN Cited by 1 source

API normalization layer (cross-provider LLM serving)¶

Pattern¶

Build an internal API surface that exposes a unified contract to feature teams, hiding per-provider differences in API shape, error codes, rate-limit behaviour, telemetry schema, and authentication. Translate provider-specific signals into a unified internal vocabulary so application logic doesn't embed any one cloud's idioms.

The canonical wiki implementation: the API normalization sub-layer of Slack's Intelligent Routing Layer (Source: sources/2026-05-28-slack-slack-ai-the-path-to-multi-cloud):

"Each provider has its own unique API patterns, proprietary error codes, and distinct rate-limiting behaviors. We had to build a robust normalization layer to ensure that a 'Rate Limit Exceeded' from one provider and a 'Throttling Exception' from another were handled identically by our application logic."

When to use it¶

Multi-cloud LLM serving — the pattern is essentially a prerequisite for multi-cloud LLM serving.
Multi-provider single-cloud — even on one cloud, when multiple model providers (Anthropic, OpenAI, Mistral, Meta) are routed through, normalisation simplifies feature code.
Per-feature optimisation desired — feature code should declare what it needs (latency budget, quality target, cost ceiling) and let the routing layer pick the best provider/model.
Internal evaluation pipeline exists — A/B testing of models is meaningless if every model swap requires feature-code changes.

When NOT to use it¶

Single provider, single model — over-engineering.
Feature teams need provider-specific capabilities — the normalisation layer might block useful provider-exclusive features unless extended.
Provider APIs are extremely volatile — keeping the layer current can be expensive.

What gets normalised¶

Slack discloses three concrete normalisation axes; two more are implicit from the routing-layer architecture.

1. API patterns¶

Request/response shapes, streaming protocols, batch APIs, async vs sync semantics. The normalisation layer exposes one internal request shape and translates to/from provider-specific shapes:

Internal contract                Provider A (Bedrock)
{                          ─▶    POST /model/.../invoke
  feature: "recap",                {body: {...}}
  prompt: "...",
  max_tokens: 1024,                Provider B (Vertex AI)
  stream: true             ─▶      POST /publishers/.../predict
}                                  {instances: [{...}]}

2. Proprietary error codes¶

Every provider has a different taxonomy. The normalisation layer maps to one internal error type set:

Bedrock: ThrottlingException     ─▶ INTERNAL: RateLimitExceeded
Vertex:  ResourceExhausted       ─▶ INTERNAL: RateLimitExceeded
Bedrock: ServiceUnavailable      ─▶ INTERNAL: ProviderUnhealthy
Vertex:  Internal                ─▶ INTERNAL: ProviderUnhealthy
Bedrock: ValidationException     ─▶ INTERNAL: BadRequest
Vertex:  InvalidArgument         ─▶ INTERNAL: BadRequest

The wiki's first canonical instance is Slack's verbatim "'Rate Limit Exceeded' from one provider and a 'Throttling Exception' from another were handled identically by our application logic."

3. Rate-limiting behaviours¶

Per-token vs per-request, RPM vs TPM, per-customer-tier vs per-account, retry-after semantics. The normalisation layer exposes one internal back-pressure / retry contract:

Bedrock: HTTP 429 + retry-after header
Vertex:  HTTP 429 + Retry-After + TPM-quota signal
            ↓
INTERNAL: BackpressureSignal { retry_after_ms, kind }

4. Telemetry / metrics (implicit)¶

TTFT, p90 latency, error rate, token cost normalised into one internal metric pipeline so the circuit breaker and routing decisions can compare providers.

5. Authentication (implicit)¶

Slack mentions "secretless authentication" as one of the cold-start engineering hurdles; the auth normalisation hides provider-specific federation flows from feature code.

Trade-offs¶

Compared to…	Wins	Loses
Direct provider SDK calls	Provider portability + unified error handling + cross-provider routing	Engineering investment up-front + ongoing maintenance
OpenAI-compatible API surface	Unified internal contract owned in-house, can normalise everything (errors, rate-limits, telemetry, auth)	More engineering work than relying on industry-de-facto standards
LangChain / SDK abstraction	Production-grade routing + circuit breaker + A/B testing built into the layer	Less flexibility per-feature than direct SDK access
Per-feature ad-hoc adapters	Targeted per-feature optimisation	Code duplication; cross-feature consistency suffers

Risks and mitigations¶

Normalisation drift — provider releases new error / API pattern and the layer doesn't catch it. Mitigation: per-provider integration tests + provider-API change monitoring + per-error-class observability.
Lossy translation — provider-exclusive capabilities expressed in the unified API may lose nuance. Mitigation: escape hatches for advanced features; explicit per-provider capability detection.
Latency overhead — translation adds cycles. Mitigation: hot-path optimisation; usually small vs network + inference latency.
Versioning — internal contract changes break feature consumers. Mitigation: backward-compatible internal schema evolution.

Composition with other patterns¶

patterns/multi-cloud-llm-serving — API normalisation is the prerequisite that makes multi-cloud routing practical.
patterns/model-fallback-hierarchy-with-circuit-breaker — the breaker depends on a unified error / health signal vocabulary that the normalisation layer provides.
patterns/circuit-breaker — composes inside; the classical pattern uses unified errors.

What's NOT in this pattern¶

Model selection logic — separate concern; the normalisation layer handles uniformity, not which model to pick.
Quality benchmarking — separate concern; the normalisation layer doesn't decide quality, just uniformity.
Workload classification — separate concern; the normalisation layer doesn't know which features are latency-sensitive vs bursty.

Seen in¶

sources/2026-05-28-slack-slack-ai-the-path-to-multi-cloud — canonical wiki disclosure of the API normalisation layer as one of five sub-systems of Slack's Intelligent Routing Layer. Verbatim Rate-Limit-Exceeded / Throttling-Exception unification example. Position in the architecture: between application features and provider-specific endpoints; sits alongside circuit breaker / metric-driven model selection / A-B routing / secretless auth as the four other subsystems.