Skip to content

PATTERN Cited by 11 sources

Central proxy choke point

Central proxy choke point is the organisational-scale posture of forcing all AI / LLM / agent traffic in an enterprise through one proxy before it reaches any provider, any MCP server, or any inference endpoint. The choke point is where identity, auth, key injection, audit logging, rate limiting, budget enforcement, fallback routing, telemetry collection, and policy enforcement all live — not in the clients.

Why "choke point" instead of just "proxy"

The discipline is architectural: there is no second path to AI capability. Clients have no direct provider credentials. Developers can't bypass the gateway by pasting an API key on a laptop — their laptop doesn't have the API key. The gateway owns every LLM call by virtue of being the only entity in the network topology that possesses the upstream credentials (concepts/byok-bring-your-own-key).

Mechanics

  • Single ingress. All coding tools / agents / applications dial one URL (the gateway).
  • Client auth via a different substrate than the upstream provider — SSO, Zero Trust JWT, workload identity, enterprise credential — so provider keys never leave the gateway.
  • Server-side key injection. Gateway re-authenticates to upstream providers with its own stored keys. Client's identity is translated into gateway-internal metadata (per-user UUID, per-tenant quota cell, cost-centre tag).
  • All policy lives here. Rate limits, budgets, audit records, fallback chains, model routing, guardrails, telemetry — single surface.

Two ingested production instances

  • Cloudflare internal stack (sources/2026-04-20-cloudflare-internal-ai-engineering-stack) — every LLM request from Cloudflare's internal tooling flows through a single Hono Worker in front of AI Gateway. Worker validates the Cloudflare Access JWT, strips client auth, injects real provider keys (cf-aig-authorization), tags requests with an anonymous per-user UUID (cf-aig-metadata) resolved from D1 + KV. Reported scale: 20.18M requests/month, 241.37B tokens, 91% frontier labs / 9% Workers AI across 3,683 users.
  • Databricks Unity AI Gateway (sources/2026-04-17-databricks-governing-coding-agent-sprawl-with-unity-ai-gateway) — all coding-agent + MCP traffic from Cursor / Codex / Claude Code / Gemini CLI funnels through Unity AI Gateway. Single SSO ("one Databricks credential for all tools, GitHub, Atlassian, others"), single audit plane (Unity Catalog), single bill (Foundation Model API + BYO external capacity), single telemetry store (OpenTelemetry → Delta tables).

Both instances describe the same architectural stance. The Databricks post frames it explicitly as an admin-productivity win: "stop switching tabs between admin consoles to control rate limits and budgets for every single coding tool."

Why the pattern scales

  • Linear review cost. Adopting a new tool is a connect-to- gateway change, not an N-vendor security review.
  • Model-swap at config-change speed. New frontier model drops weekly — gateway config change, no redeploy of any client.
  • Per-identity quotas become portable. Budgets follow the developer, not the tool (patterns/unified-billing-across-providers).
  • Telemetry cardinality is bounded. One schema, one store, one dashboard — joinable with business data (patterns/telemetry-to-lakehouse).

Costs / risks

  • Single point of failure. Gateway-down = AI-down for the whole org. Mitigation: fallback chains inside the gateway, HA deployment, regional failover.
  • Gateway-cap-ex concern. Gateway itself becomes a hot-path component with org-scale throughput requirements.
  • Client-side escape hatches. Any client that can route around the gateway (e.g. personal-key in a dev-laptop env file) breaks the choke-point invariant. The pattern only works when clients cannot reach upstream providers directly.
  • Governance-centralisation downsides. One policy surface means one team owns LLM policy. Teams wanting experimentation friction usually want less gateway, not more.

Relation to other patterns

Seen in

  • sources/2026-09-30-cloudflare-identify-ai-model-overuse-with-user-insights — the choke point as a corpus for out-of-band analysis. Because all AI traffic already flows through AI Gateway, Cloudflare can run a dedicated categorization Worker asynchronously over the proxied logs — classifying each conversation's task category, complexity, ambiguity, stakes, and context-dependence after the response is served (User Insights model overkill / task / turns views). The choke point's value here is not inline enforcement but complete, joinable log coverage: only the single vantage point that sees every request can classify usage org-wide, and doing it off the request path keeps the proxy's hot path latency-free (analysis trails by ~1 day). The same signal then feeds the inline Auto Router — the choke point hosts both the offline observe and inline act halves of model-fit routing. (Source: sources/2026-09-30-cloudflare-identify-ai-model-overuse-with-user-insights)

  • sources/2026-09-29-cloudflare-adaptive-application-security-for-the-ai-era-how-cloudflare — the inline reverse proxy as the enforcement point for the whole application-security framework. Cloudflare's edge sits inline on >20% of the web, and "because Cloudflare is inline, we can turn those insights into protections immediately." Every stage runs through this choke point: the WAF reverse proxy "protects applications at runtime by filtering traffic before it reaches the origin," running positive security → ML/LLM-tactic detection → business-logic protection → real-time threat intel as ordered layers at one point. The framework's power is that the choke point also combines reverse- proxy + forward-proxy (Cloudflare One) visibility to correlate external attacks with internal lateral movement.

  • sources/2026-08-14-cloudflare-secure-all-your-internal-vibe-coded-applications-in-one-click — dispatch Worker as choke point for private-by-default platforms. In Workers for Platforms, all namespace traffic passes through a single dispatch Worker; setting a Access policy there makes every tenant Worker private by default (account-wide-default-policy) — the auth choke point is the dispatcher, configured once and inherited by all deployments.
  • sources/2026-08-14-cloudflare-how-cloudflare-detects-mcp-traffic-and-helps-secure-it — network-SWG choke point for MCP. The Cloudflare Gateway secure web gateway is the single vantage point that classifies MCP traffic at the protocol layer (protocol-header-traffic-classification), attributes it to users/servers, and enforces Portal-only egress. Distinct altitude from the AI-Gateway LLM-proxy instances of this pattern — here the choke point is the device→internet network path, not the app→model path — but the same posture: agents that route around the choke point escape every layer of policy.
  • sources/2026-05-20-databricks-governing-ai-agents-at-scale-with-unity-catalog — Generalises the choke-point shape from coding-agent scope (2026-04-17) to org-wide agent populations. The 2026-05-20 Databricks four-pillars post extends Unity AI Gateway from "every coding tool" to "every model call, every tool invocation, every agent interaction" across dev / analytics / sales-ops / support / marketing / finance. The choke-point posture acquires four named layers attached to the same proxy: Service Policies (per-tool-call admission via UC functions), Guardrails (inline content scanning, fail-closed), Inference Tables (full-payload audit writes to lakehouse), Budgets (per-user / per-group thresholds with alerts). Re-frames the choke-point property as the enabling discipline for governance-travels-with-resources — agents that bypass the gateway escape every layer of policy. Also canonicalises three-layer-agent-control (permissions / Service Policies / Guardrails) as the load-bearing composition the choke-point hosts.
  • sources/2026-04-16-cloudflare-ai-platform-an-inference-layer-designed-for-agents — catalog-scale choke point. The 2026-04-16 AI Platform post productises one env.AI.run() binding against 70+ models across 12+ providers; the gateway is the only vantage point that sees aggregate spend, per-tenant attribution, and cross-provider failover. Extends the pattern with buffered resumable streaming (buffered-resumable-inference-stream) — the choke-point now also owns stream lifetime, not just request routing.
  • sources/2026-04-20-cloudflare-internal-ai-engineering-stack — Hono-Worker-in-front-of-AI-Gateway; scale numbers disclosed.
  • sources/2026-04-17-databricks-governing-coding-agent-sprawl-with-unity-ai-gateway — Unity AI Gateway specialised for coding-agent + MCP governance.
  • sources/2026-04-20-cloudflare-orchestrating-ai-code-review-at-scale — CI-embedded choke point for AI code review. Every LLM call from Cloudflare's AI Code Review system flows through AI Gateway — the coordinator, all seven sub-reviewers, and re-review retries — all under one set of keys, one unified spend view, one failback topology. The reviewer-config KV Worker is the choke-point's control-plane surface: flip a provider's enabled flag and every running CI job re-routes within 5 seconds without a code deploy. Adds the CI- integration variant to the pattern — choke-point + per-role remote routing + per-tier circuit breakers.
  • sources/2026-03-19-pinterest-building-an-mcp-ecosystem-at-pinterest — Enterprise-internal MCP choke point. Pinterest's MCP Registry + Envoy mesh validation together form the choke-point for MCP traffic inside Pinterest. "Envoy validates the JWT, maps it to X-Forwarded-User, X-Forwarded-Groups, and related headers, and enforces coarse-grained security policies." Differs from the Cloudflare/Databricks instances on two axes: (a) internal-employee traffic not external-LLM-API traffic — the choke-point's upstream-credential job is absent (Pinterest isn't injecting API keys); (b) the choke-point's policy surface is Envoy config driven by registry-backed security review outcomes, not a dashboard flag. Shares with the other instances: "there is no second path to AI capability" — MCP tools are only reachable through the Envoy + registry substrate. Canonical wiki instance of the enterprise-SSO-piggyback choke-point shape (layered-jwt-plus-mesh-auth).
  • sources/2026-04-14-redpanda-openclaw-is-not-for-enterprise-scale — Agent-workforce-scale choke point framing with kill switch as a first-class primitive. Redpanda's 2026-04-14 Openclaw is not for enterprise scale post canonicalises the choke-point pattern as component #1 of the four- component agent production stack (Gateway + Audit log + Token vault + Sandboxed compute). Verbatim: "The gateway boils down to having a single choke point for all agentic access to external systems and information. It allows you to have full observability into what your agents are doing [...] A gateway is also a centralized place for enforcing rate limits and applying guardrails." The post adds a kill-switch capability to the canonical choke-point capability set — "the kill switch goes to turn off a rogue agent leaking your Salesforce customer data. No need to hunt down API access for 27 different services and systems, just turn it off for a single service or set of services for your entire digital workforce at once." Distinct from the Cloudflare / Databricks / Pinterest instances on two axes: (a) agent traffic as the primary abstraction (not LLM-inference traffic or MCP traffic specifically, but the agent's full egress surface); (b) composed with the concepts/token-vault as a separate architectural component — the gateway doesn't hold credentials, it asks the vault per-call. This is the rhetorical-voice essay altitude (no scale numbers, no mechanism depth), but the "Gateway + Audit trail + Token vault + Sandboxed compute = Agents in production" formula canonicalises the choke-point as the keystone of a four-component production-minimum.

Identity-linked behavioral analytics (Cloudflare, 2026-08-05)

A gateway's telemetry is materially stronger when its principal is verified at ingress rather than inferred from a shared provider key. Cloudflare places Access ahead of AI Gateway, adds the verified cf.user_id to request metadata, then uses the same choke point for per-user controls and User Insights. The feature scores session cost against the account's rolling p95 and requires an account-wide p99 floor before surfacing an anomaly. It is a monitor-only review signal, not inline enforcement — an important boundary between a central proxy's evidence role and its authorization role. (Source: sources/2026-08-05-cloudflare-catching-rogue-ai-behavior-with-identity-aware-analytics)

Merged aliases

  • enforcement-throttler-proxy- gateway-centralized-tool-credential-management
Last updated · 766 distilled / 2,225 read