Skip to content

PATTERN Cited by 2 sources

Two-tier connection pooling

What it is

Two-tier connection pooling is the architectural pattern where connections between application code and a database pass through two distinct pools with different responsibilities:

  1. Application-tier pool — owned by the application process (e.g. HikariCP in JVM, database/sql pool in Go, ActiveRecord::ConnectionPool in Rails). Eliminates the per-request connection-handshake cost for this process.
  2. Proxy-tier pool — owned by a central proxy between applications and the database (e.g. Vitess VTTablet, pgbouncer, ProxySQL). Enforces the database's max_connections / memory budget once, globally, across all application processes.

The pattern separates two failure modes that a single pool conflates: per-request latency (fixed by app-tier pool) and global admission control (fixed by proxy-tier pool).

Problem

A single-tier pool (just the app-tier pool) is the default. At small scale (one app process, one DB), it works: the pool is sized conservatively to stay under max_connections, and the handshake cost is eliminated.

Problems appear as soon as N > 1 application processes share the database:

  • Each process has its own pool of P connections → aggregate is N × P upstream connections.
  • The DB's max_connections cap is breached at N = max_connections / P. For RDS's 16,000 cap and P = 20 (typical), this is 800 application processes — reached trivially in any medium-scale deployment.
  • Serverless compute makes N explode: every function invocation is a fresh process with a fresh pool. N is the peak concurrent function count, not the number of static server nodes.

Raising max_connections isn't a solution because of memory overcommit risk — crossing the memory envelope exposes the database to OOM crashes.

Solution

Interpose a proxy pool between applications and database:

┌─────────────┐   ┌─────────────┐   ┌─────────────┐
│  App pool   │   │  App pool   │   │  App pool   │      ← app-tier
│  N app1     │   │  N app2     │   │  N app3     │
└──────┬──────┘   └──────┬──────┘   └──────┬──────┘
       │                 │                 │
       └─────────────────┼─────────────────┘
                         ▼
                 ┌───────────────┐
                 │   PROXY POOL  │                         ← proxy-tier
                 │  (VTTablet /  │
                 │  pgbouncer /  │
                 │  Global Route │
                 │   Infra)      │
                 └───────┬───────┘
                         │
                         │  small capped
                         │  upstream pool
                         ▼
                 ┌───────────────┐
                 │   DATABASE    │                         ← origin
                 │ max_connections│
                 └───────────────┘

Each tier serves a different constraint:

  • App-tier pool: per-process pool. Size = peak concurrent queries the process will execute. Purpose: eliminate the cold-start connection-handshake cost (~50 ms MySQL SSL handshake per new connection — see ).
  • Proxy-tier pool: shared pool. Size = max_connections on the database. Purpose: enforce the memory-safe ceiling once, globally, and convert many client-facing connections into a capped number of upstream connections.

Canonical instances

Vitess VTTablet (in-cluster pool)

VTTablet is the pre-sharding Vitess component that owns the MySQL connection pool for each shard primary. Applications connect to VTGate → VTTablet; VTTablet holds the MySQL connections. The pool is lockless, using atomic operations and non-blocking data structures (see ).

PlanetScale Global Routing Infrastructure (edge pool)

PlanetScale stacks a third tier on top of VTTablet: a CDN- like edge pool that terminates the client's MySQL session at the edge node nearest the client and backhauls queries to VTTablet over warm, long-held, multiplexed backhaul connections (see systems/planetscale-global-network). This is the specific architectural substrate that enables the benchmarked 1M-concurrent-connections ceiling (source: ), 62.5× above RDS MySQL's 16k-instance cap (source: ).

pgbouncer / ProxySQL (generic pooler tier)

The pattern also applies to vanilla Postgres / MySQL: pgbouncer in front of Postgres, ProxySQL in front of MySQL, accepting many client connections and multiplexing to a small upstream pool. No edge tier; just application pool + proxy pool.

When to use

  • Application is horizontally scaled across many processes / containers / functions.
  • Serverless / edge deployment: cold-start makes app-tier pooling per-process necessary, but function-count multiplies quickly.
  • Multiple different applications share the same database (canonicalised in : "Once an application horizontally scales from one server to 'hundreds or thousands,' and once multiple applications share the same database, application-level pools can't enforce a global connection ceiling").
  • Database has a memory-derived max_connections that cannot be raised without adding memory.

When NOT to use

  • Single application, small scale: the proxy tier is overhead for no benefit. App-tier pool alone is enough.
  • Session-level state heavy: proxy-tier pools historically fought this poorly (see concepts/tainted-connection +

for Vitess's three-era evolution). Modern proxy pools (Vitess v15+ settings pool) handle it, but it adds complexity. - Transaction-heavy workloads where transactions span many statements: proxy-tier pools in transaction mode keep the connection pinned for the transaction duration, which reduces pool efficiency. Proxy-tier pools in statement mode don't work for transactions at all.

Trade-offs

  • Extra proxy hop adds RTT (typically sub-millisecond for same-region VTTablet / pgbouncer; edge-tier adds more depending on client locality).
  • Complexity: two systems to operate, monitor, alert on.
  • Debugging: a misbehaving query goes through two pools — observability must trace across both tiers.
  • Session-state semantics at the proxy tier are tricky (see concepts/tainted-connection).

The benefits at scale dominate: the pattern is what enables benchmarked ceilings 62.5× the single-database ceiling (1M vs 16k) with no OOM risk.

Seen in

  • — canonical benchmarked demonstration: VTTablet pool + Global Routing Infrastructure pool sustain 1M concurrent connections against a PlanetScale database; Liz van Dijk explicitly names the two-tier framing ("Vitess and PlanetScale offer connection pooling on the VTTablet level … In addition to that, PlanetScale's Global Routing Infrastructure provides another horizontally scalable layer of connectivity").
  • — canonical mechanism detail on the in-cluster tier; three-era VTTablet pool-design evolution with the ~50 ms MySQL SSL handshake cost as the why of the app-tier pool.
  • — canonical contrast: RDS MySQL's 16k instance cap is the ceiling the two-tier architecture is designed to exceed; Reyes names the PlanetScale architectural answer without specifying.
  • — client-side complement surfaced. Matthieu Napoli (2023-05-03) benchmarks the alternative shape: instead of fixing the serverless-DB-connection problem at the server-side proxy tier (this pattern), deploy a persistent-process request handler (Laravel Octane via Bref) inside each Lambda execution context so the DB connection + TLS handshake + application bootstrap are amortised across BREF_LOOP_MAX = 250 invocations per execution context. Recovers 5.4× p50 (75 ms → 14 ms) against PlanetScale — see patterns/persistent-process-for-serverless-php-db-connections. The two patterns are composable: PlanetScale's two-tier proxy pool absorbs upstream connection churn while Octane eliminates client-side per-request handshake overhead.
  • — pedagogy-101 altitude canonical statement of the two-tier shape. Brian Morrison II (PlanetScale, 2022-10-21): "Vitess takes the lightweight connections established by each client to VTGate and maps them to a smaller pool of MySQL connections managed by VTTablet. This process in turn helps to avoid overloading the individual MySQL processes, resulting in lower resource utilization since only VTTablet needs to connect to the underlying MySQL process." The canonical first-principles framing that the quantitative sibling posts (, ) measure and elaborate. Names Go + gRPC + goroutines as the implementation-language substrate that makes cheap-per-client connections possible ("With the concurrency features built into the Go language, Vitess is able to easily handle thousands of clients simultaneously") — the why behind the 1M-connection ceiling being three orders of magnitude higher than bare MySQL's 16k.
Last updated · 542 distilled / 1,571 read