EXPEDIA 2026-03-24 Tier 3

Expedia — Operating Trino at Scale With Trino Gateway¶

Summary¶

Expedia Group's data-platform team writes up their production use of Trino Gateway — an open-source proxy / load balancer for Trino clusters (originally forked from Lyft's Presto Gateway and integrated into the Trino ecosystem) — and the four user-experience features they contributed upstream. The architectural thesis: at scale, pointing users directly at individual Trino cluster endpoints stops working (too many clusters, too many workload types, too much user-side coordination); a single-endpoint gateway that routes queries to the most appropriate backend cluster (based on the target tables, query text, or source application) becomes the load-bearing abstraction. On top of that, the gateway absorbs concerns that were previously awkward for Trino operators — no-downtime cluster upgrades (blue/green or canary swap behind the gateway), cluster-health-aware routing (HEALTHY / UNHEALTHY / PENDING states computed by the gateway and honored by the RoutingManager), and unified query-history inspection across the fleet. Expedia's contributions are operator-UX features: view and edit routing rules in the UI (no config-file edits), add a source filter to the query-history page, display per-cluster health on the cluster page, and open full (previously 200-char-truncated) query text in a separate window.

Key takeaways¶

At scale, a Trino fleet stops being addressable as individual cluster endpoints. As organisations scale their analytics platforms, they hit "increased query complexity, higher concurrency, and the need for specialised cluster configurations." Directing users at specific cluster URLs stops being practical as the user base grows. A gateway in front of the fleet lets users point one connection URL at the whole cluster fleet and have the gateway decide which backend cluster best fits each query. (Source: this post)
The canonical Trino cluster-segregation pattern is three workload shapes: Adhoc / ETL / BI. Expedia names this as "a common pattern for organizations using Trino at scale":
Adhoc clusters — mixed workloads, medium concurrency, varied query complexity; exploratory analysis + development.
ETL clusters — high-volume, high-complexity queries, low concurrency; data integration / transformation / cleansing / enrichment.
BI clusters — low-complexity queries, high concurrency; dashboards and report tools like Tableau and Looker, often against pre-aggregated data. Each cluster configuration is tuned to the shape of its workload (query size vs. concurrency ceiling) rather than trying to balance both on one cluster. (Source: this post)
Workload-aware routing is the gateway's primary architectural contribution. Routing rules inspect the incoming query — the tables being queried, the query body text, HTTP headers like X-Trino-Source — and decide which routingGroup (cluster pool) should execute it. The post lists three production-relevant rule shapes: (a) large-table isolation — route queries touching named large tables to a heavy-workload cluster so smaller queries are not queued behind them; (b) metadata-query offload — route lightweight health-check-style queries (select version(), show catalogs) that BI tools run constantly to a single-node metadata cluster so dashboard extract-failure rates drop and user-level limits can be tuned independently; (c) BI-source routing — detect queries coming from Tableau / Looker (via X-Trino-Source header) and send them to BI-optimised clusters even if users did not change their client config. (Source: this post)

Routing rules are written as short Groovy-style condition + action scripts evaluated per query. The structure is name + description + condition (boolean; "true" matches all) + actions (script body that inspects the query and writes a routingGroup into the result map). Rule authors have access to trinoQueryProperties (parsed query: getTables(), getBody()) and request (HTTP: getHeader()). Canonical example bodies from the post:

foreach (table : trinoQueryProperties.getTables()) {
  String tableSuffix = table.getSuffix();
  if (tableSuffix.contains("table1") || tableSuffix.contains("table2")) {
    result.put("routingGroup", "large-cluster");
    return;
  }
}

if (trinoQueryProperties.getBody().toLowerCase().contains("select version()")
    || trinoQueryProperties.getBody().toLowerCase().contains("show catalogs")) {
  result.put("routingGroup", "metadata-cluster"); return;
}

condition: 'request.getHeader("X-Trino-Source") contains "Tableau"
         || request.getHeader("X-Trino-Source") contains "Looker"'
actions:   - 'result.put("routingGroup", "gateway-bi")'

This is the canonical routing-rules-as-config shape: human-readable, per-rule, hot-editable. (Source: this post)

Before Expedia's contribution, routing rules were managed by editing config files directly. Rule inspection required reviewing configuration files or examining the gateway environment; there was no UI surface to show what rules were currently active. Both sides — "what rules exist?" and "change this rule" — were unfriendly to administrators. Since the routing rules are "the crux of the gateway" (they are the decision function that picks the backend cluster), this was a material operator-UX gap. Expedia's PR #433 added a UI to view and modify routing rules directly; changes persist when shared storage is configured. (Source: this post)
The gateway enables no-downtime cluster upgrades (blue/green or canary). With users addressing a single gateway URL, an operator can add a new (upgraded) Trino cluster to the fleet, shift traffic via routing rules (or routingGroup membership) from the old to the new cluster, and drain the old one — all without breaking the user connection URL. The post lists "no-downtime upgrades for Trino clusters behind the gateway in a blue/green model or canary deployment model" as one of the four headline gateway advantages. Capacity changes (adding or removing clusters) have the same property — "transparent change of capacity of Trino clusters without user interruptions." (Source: this post)
Cluster-health is a first-class routing input with three states. Before Expedia's contribution, the gateway UI only offered an active/inactive toggle for each cluster, which is the operator's intent rather than the cluster's current status. The upgraded UI (Expedia's PR #601) displays the health as computed by the RoutingManager:
HEALTHY — cluster health-checks report the cluster is ready; the RoutingManager routes requests to it.
UNHEALTHY — cluster health-checks report unhealthy; the RoutingManager does not route requests to it.
PENDING — cluster is starting up; treated as unhealthy (no routing) until it crosses into HEALTHY. This three-state model (concepts/cluster-health-check) generalises beyond Trino — the same HEALTHY / UNHEALTHY / PENDING trichotomy applies to any load-balanced backend fleet where the LB needs to distinguish "starting up" from "broken" because both yield "do not route" but they do so for different reasons and at different timescales. (Source: this post)
Query history unification across the fleet is a secondary gateway win. Because every query goes through the gateway, the gateway knows about every query — not just the queries on any one cluster. This lets the gateway provide a single history pane across the entire Trino fleet. Expedia added two usability improvements here: PR #551 added a Source filter to the history page (filter by the client application that initiated the query — "quickly isolate queries from specific applications, making it much easier to diagnose issues"); PR #740 removed the previous 200-character query-text truncation and added an "open full query text in a separate window" affordance so users don't have to visit the originating cluster UI to see the full query. (Source: this post)
Four headline advantages of the gateway are stated explicitly in the post: (a) "Use of a single connection URL for client tool users with workload distribution across multiple Trino clusters"; (b) "Automatic routing of queries to dedicated Trino clusters for specific workloads or specific queries and data sources"; (c) "No-downtime upgrades for Trino clusters behind the Gateway in a blue/green model or canary deployment model"; (d) "Transparent change of capacity of Trino clusters without user interruptions." (Source: this post)

Systems¶

systems/trino-gateway — the open-source proxy / load balancer in front of one or more Trino clusters. Provides a single connection URL for clients, routes queries to the appropriate backend cluster via routing rules, and centralises administrative surfaces (routing-rule management, query history, cluster health). Originated at Lyft as Presto Gateway; forked and integrated into the Trino ecosystem with contributions from various organisations including Expedia. Key architectural components named: the RoutingManager (consults cluster health and routing rules, decides which cluster handles a given query) and the routing-rule engine (per-rule condition + actions evaluated against trinoQueryProperties + request).
systems/trino — the distributed SQL query engine the gateway fronts. "A fork of PrestoSQL"; "fast, scalable insights without requiring data relocation". Expedia runs multiple Trino clusters segregated by workload shape (Adhoc / ETL / BI) and sizes each cluster for its shape rather than trying to cover all three on one cluster.
systems/presto — the predecessor engine; named explicitly ("Trino — a fork of PrestoSQL") and as the origin of the gateway concept ("originated at Lyft as Presto Gateway, serving as a proxy and load balancer for PrestoDB"). The Lyft origin is historically load-bearing: the gateway predates Trino, and the Trino ecosystem inherited the pattern by forking the Lyft code.
systems/tableau — named as one of the BI tools that BI-source routing rules detect (via the X-Trino-Source header) and route to BI-optimised clusters.
systems/looker — named alongside Tableau as the other canonical BI tool whose queries the post routes to BI clusters.

Concepts¶

concepts/workload-aware-routing — the architectural thesis of the post: at scale, a query gateway should make routing decisions based on the shape of the query (tables touched, query body, source application) and land it on the cluster whose configuration best fits that shape. Contrast with round-robin / least-connections LB, which treats all backends as interchangeable. Workload-aware routing assumes the backend fleet is deliberately heterogeneous and the router's job is to match query shape to cluster shape.
concepts/single-endpoint-abstraction — users point at one URL; the gateway does the cluster selection. Enables no-downtime upgrades (backend substitution is invisible), capacity changes, and routing-rule updates. This is the same abstraction principle as Envoy or ECS Service Connect sitting in front of a service fleet, applied to a SQL-query engine fleet.
concepts/cluster-health-check — HEALTHY / UNHEALTHY / PENDING trichotomy computed by the gateway and honored by the RoutingManager. PENDING is the load-bearing state: it distinguishes "starting up" (expected to become HEALTHY) from "UNHEALTHY" (broken); both are "do not route" but they emit different alerts and mature on different timescales.

Patterns¶

patterns/query-gateway — the architectural pattern of placing a proxy in front of a heterogeneous fleet of SQL-query clusters, giving clients a single connection URL, and having the proxy route each query to the cluster best suited for it. Trino Gateway is the canonical instance; Presto Gateway was the precursor. Generalises beyond Trino — any fleet of query engines with heterogeneous cluster shapes can adopt the same pattern.
patterns/workload-segregated-clusters — running dedicated clusters per workload shape (Adhoc / ETL / BI) instead of a single "one-size-fits-all" cluster. Each cluster's config is tuned to its workload's concurrency + query-complexity profile. The gateway is the necessary complement — without a gateway, users would have to know which cluster to target; with a gateway, segregation is invisible to users.
patterns/routing-rules-as-config — routing decisions encoded as human-readable, individually-named rules with condition + actions + access to parsed query properties and HTTP request context. Hot-editable (with UI support post-Expedia contribution) and persistable via shared storage. Decouples "which cluster?" from application code and from cluster config.
patterns/no-downtime-cluster-upgrade — blue/green or canary cluster swap behind a gateway. The gateway absorbs the user connection URL, the operator spins up a replacement cluster alongside the old one, routing rules shift traffic, then the old cluster is drained. Listed in the post's four-point advantages.

Operational numbers¶

No production numbers disclosed:

No query volume / QPS / cluster count.
No per-cluster sizing (node count, memory, CPU).
No concurrency ceilings for ETL / BI / Adhoc shapes.
No routing-rule evaluation latency.
No fleet-upgrade cadence or rollout pattern details.
No extract-failure-rate before/after for the metadata-cluster routing rule (only "reduce extract failure rates" qualitative).

The post is a feature / contribution retrospective, not a production-scale retrospective. The architectural claims (workload segregation, routing rules, cluster health) are independent of the numbers.

Caveats¶

Feature-centric framing. The post is organised around four UI-level contributions Expedia made; the "why operate Trino at scale with a gateway?" material is background framing, not the main subject. Readers looking for deep operational lore (incident walkthroughs, cluster right-sizing methodology, per- workload SLA ownership) won't find it here.
No comparison to alternatives. The post does not evaluate Trino Gateway vs. alternatives like Tabby competitors, generic HTTP LBs with custom logic, or cloud-managed Trino offerings' own routing. Readers should not conclude "Trino Gateway is the right choice" from this post alone — only that "if you use Trino Gateway, these four UX features will be available."
Routing-rule language is Groovy-like but not explicitly named. The snippets look like Groovy or a Groovy-like scripting surface, with access to typed helpers (trinoQueryProperties, request). The post does not name the language or link to its grammar; rule authors should consult the Trino Gateway docs.
"Shared storage" for rule persistence is not detailed. The post says "changes are persisted when a shared storage is used" but does not name the storage backend (presumably the same MySQL / Postgres / H2 backend Trino Gateway supports for its metadata, per the project's own docs).
Health-check mechanics are not described. The post names the three states (HEALTHY / UNHEALTHY / PENDING) and the outcome (the RoutingManager only routes to HEALTHY) but not the mechanism (what endpoint on the Trino cluster is probed, how often, what failure threshold flips a state, how flapping is handled). Operators should consult the Trino Gateway source or docs for the probe implementation.
"Tier-3 source, feature-driven post." This article passes the Tier 3 content filter on architectural-substance grounds (distributed SQL engine infrastructure, production workload segregation, heterogeneous-cluster gateway design) despite its promotional framing of Expedia's upstream contributions.

Raw¶

Raw file: raw/expedia/2026-03-24-operating-trino-at-scale-with-trino-gateway-5217f9b0.md
Original URL: https://medium.com/expedia-group-tech/operating-trino-at-scale-with-trino-gateway-41824af788de
Project repo: https://github.com/trinodb/trino-gateway
Contributed PRs referenced: #433 (UI for routing rules), #551 (source filter on history page), #601 (cluster health display), #740 (full-query-text window)