Skip to content

PLANETSCALE 2026-04-02

Read original ↗

PlanetScale — Patterns for Postgres Traffic Control

Summary

Josh Brown's 2026-04-02 PlanetScale post is the canonical Go-language implementation companion to the two earlier Traffic Control posts — the 2026-04-11 queue-health post (mixed-workload contention framing) and the 2026-03-31 Ben Dicken graceful-degradation post (user-perceived-priority framing). Where those two posts framed what Traffic Control does and why, Brown's post canonicalises how to wire it up in a typical Go web service: five composable tagging patterns that each target a distinct failure mode (service-isolation, route-isolation, deployment-canary, SaaS tier, background jobs) + two operational-integration patterns (Enforce- mode SQLSTATE 53000 error handling, Warn-mode [[concepts/postgres-notice- warning-channel|OnNotice]] observability) + a layered-composition framing showing that all five axes can be active simultaneously and are AND-ed by the Traffic Control engine.

The load-bearing architectural move is the context.Context- threaded SQLCommenter tag propagation pattern — canonical new patterns/context-threaded-sql-tag-propagation — which is the Go-idiomatic counterpart to the ORM-middleware pattern canonicalised on patterns/query-comment-tag-propagation-via-orm from the 2022 Coutermarsh + Ekechukwu Rails post. Go has no global request-local storage idiom comparable to Rails's CurrentAttributes or thread-local; instead, tags ride on context.Context through every function that touches the database, and a wrapper QueryContext / ExecContext method renders them to SQL at the last possible moment.

Key takeaways

  1. Two application-side helpers are load-bearing substrateappendTags(query, tags) → taggedQuery (deterministic sort + URL-encode + SQLCommenter format) + tagsFromContext( ctx) → map[string]string (copy-on-read for thread-safety). Every pattern in the post layers on top of these two primitives. Verbatim canonical shape:

    func appendTags(query string, tags map[string]string) string {
        if len(tags) == 0 { return query }
        parts := make([]string, 0, len(tags))
        for k, v := range tags {
            parts = append(parts, fmt.Sprintf("%s='%s'", k, url.QueryEscape(v)))
        }
        sort.Strings(parts) // deterministic order
        return query + " /*" + strings.Join(parts, ",") + "*/"
    }
    
    Deterministic ordering matters because the Postgres plan cache is comment-bytes-sensitive in some configurations — the same logical tag set must produce the same SQL text.

  2. Service-isolation via Postgres username + application_namePattern 1: the coarsest isolation axis. A dedicated Postgres role per microservice makes username='pscale_api_ 123abc' available to Traffic Control as a budget key; application_name=myapp in the connection string (or set via ParseConfig / SetRawConnect on the driver) is a second, orthogonal axis. "This also helps in incident response: you can immediately cap a service's resource share without redeploying anything." The connection-string axis is load-bearing — it works even if the application emits zero SQLCommenter tags because Postgres itself (not the app) is the source of truth for the tag value. Canonicalises the three auto-populated SQLCommenter tags already canonicalised via the 2026-03-24 enhanced-tagging-in-insights post (application_name, username, remote_address) as the baseline attribution surface that's available before any app-level discipline is deployed.

  3. Route-isolation via HTTP middlewarePattern 2: the /api/export CSV-report endpoint should not be able to kill the /api/checkout flow. Canonical new patterns/route-tagged-query-isolation: an HTTP middleware injects route=<r.Pattern> + app=web into the request context, and wrapper QueryContext/ExecContext methods render them. Verbatim canonical implementation:

    func SQLTagMiddleware(next http.Handler) http.Handler {
        return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
            tags := tagsFromContext(r.Context())
            route := strings.ReplaceAll(strings.ReplaceAll(r.Pattern, "{", ":"), "}", ":")
            tags["route"] = route
            tags["app"] = "web"
            ctx := contextWithTags(r.Context(), tags)
            next.ServeHTTP(w, r.WithContext(ctx))
        })
    }
    
    The { / }: replacement is because Go 1.22+ HTTP patterns use {param} placeholders; Traffic Control rule strings don't benefit from the brace syntax. Operational payoff: "the violation graph in Traffic Control will show you exactly which route tag is hitting limits."

  4. Deployment / canary-tag via startup environment variablePattern 3: DEPLOYMENT_TAG=new_checkout_v2 (or a git SHA 96e350426) is read at startup and injected into every query's tags on the new pods. "Traffic Control can then have a budget on feature='new_checkout_v2' in Warn mode from day one, so you see exactly how the new code behaves before it causes problems." Canonical deployment-gating workflow: tag new code → observe in Warn mode → decide to enforce or remove the budget after soak. Distinct from a runtime feature-flag pattern in the second half of the section (flags.Enabled(ctx, "new_order_flow")tags["feature"] = "new_order_flow") which is flag-gated, not deployment-gated. Both specialise the general [[concepts/warn-mode-vs-enforce- mode|Warn mode → Enforce mode]] lifecycle to the ship-new- code axis.

  5. SaaS tier-isolation via authentication middlewarePattern 4: the free-tier user's expensive dashboard must not degrade the enterprise customer's experience. Canonical new patterns/tier-tagged-query-isolation: after the auth middleware resolves the authenticated user, inject the user's subscription tier into the SQL tags:

    func AuthMiddleware(users *UserService, next http.Handler) http.Handler {
        return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
            user, err := users.Authenticate(r)
            if err != nil { http.Error(w, "unauthorized", 401); return }
            ctx := WithUserTier(r.Context(), user.Tier)
            next.ServeHTTP(w, r.WithContext(ctx))
        })
    }
    
    Two budgets: tier='free' (conservative) + tier='pro' (moderate); enterprise unbudgeted or high-budget ceiling. Composes with Pattern 2tier='free' AND route='api- export' is a stricter-than-tier='free'-alone combination. Canonical wiki datum: tier + route is a two-axis budget, ANDed, giving enterprise-export more headroom than free- export.

  6. Background-job isolation via dedicated connection pool + application_namePattern 5: long-running background workers use their own *sql.DB pool with application_name= background-jobs set in code (not env var), and SetMaxOpenConns(4) to cap concurrency at the connection- pool layer (a second orthogonal cap to Traffic Control's concurrent-worker dial). One-off scripts set application_ name=script-<scriptName> (e.g. script-backfill-order- totals) via the same helper. Operational workflow: run a Warn-mode budget before the job next runs, observe typical consumption, switch to Enforce at a level where a runaway job can't crowd out interactive traffic. Canonical new patterns/dedicated-application-name-per-workload: "Setting application_name on the connection string level in code ensures that it is always set for this service, no matter the query or connection string given."

  7. Enforce-mode error-class handling via SQLSTATE 53000Pattern 6: when a query exceeds its budget in Enforce mode, Postgres returns SQLSTATE 53000 with message prefixed [PGINSIGHTS] Traffic Control:. Canonical new concepts/sqlstate-53000-traffic-control-error. Verbatim pgx/v5 detection helper:

    const sqlstateTrafficControl = "53000"
    func isTrafficControlError(err error) bool {
        var pgErr *pgconn.PgError
        return errors.As(err, &pgErr) && pgErr.Code == sqlstateTrafficControl
    }
    
    Right response is query-role-dependent: analytics / reporting → return 503 Service Unavailable or cached result ("exactly the controlled failure mode Traffic Control is designed to create"); critical paths → short retry with exponential backoff. The queryWithBackoff helper shown: 100ms → 200ms → 400ms over 3 attempts, bailing out on non-Traffic-Control errors and on ctx.Done().

  8. Warn-mode observability via pgx/v5 OnNotice handlerPattern 7: in Warn mode, queries succeed but Postgres emits a notice (not an error) to the driver containing [PGINSIGHTS] Traffic Control: …. Canonical new concepts/postgres-notice-warning-channel. Canonical shape:

    config.OnNotice = func(c *pgconn.PgConn, notice *pgconn.Notice) {
        if strings.Contains(notice.Message, "[PGINSIGHTS] Traffic Control:") {
            log.Printf("traffic control warning: %s", notice.Message)
        }
    }
    
    Canonical wiki datum: the Postgres wire protocol's NoticeResponse frame is the piggyback channel that lets Traffic Control observe in-band in Warn mode without user-facing impact. "Collect these logs for a few hours of representative traffic before switching to Enforce. The pattern of which rules fire and how often tells you whether your limits need adjustment." Instrumentation altitude: the driver, not the application — every query's warn events flow through one callback.

  9. Layered-composition framing — the full canonical wire-up composes all five tag-axes in one middleware stack:

    var handler http.Handler = mux
    handler = SQLTagMiddleware(handler)          // Pattern 2: route
    handler = AuthMiddleware(s.users, handler)   // Pattern 4: tier
    jobDB, _ := newJobDB(dsn)                    // Pattern 5: jobs
    // DEPLOYMENT_TAG env var set in deployment manifest    // Pattern 3
    // dedicated Postgres role username per service         // Pattern 1
    
    Canonical new concepts/composable-tag-axes: five orthogonal axes (service / route / deployment / tier / workload), ANDed at enforcement time, "A budget on tier='free' covers all free- tier traffic regardless of route. A budget on route='api- export' AND tier='free' covers a specific combination. Multiple matching budgets all apply simultaneously and queries must satisfy every budget they match. You can build layered policies without complicated rule logic."

  10. The load-bearing adoption discipline: "Start in Warn mode, observe which budgets would fire during normal load, tighten the limits until only pathological cases trigger violations, then switch to Enforce." Same four-step flow as the 2026-03-31 graceful-degradation post (comment → warn → monitor → enforce), but now with the Go-side telemetry- collection mechanism (OnNotice log → metric → dashboard) explicitly specified. "The difference between a database outage and a degraded experience often comes down to whether you've decided in advance which traffic to shed. Traffic Control makes that decision explicit and configurable instead of leaving it to whichever query happens to win a resource race."

Systems extracted

  • PlanetScale Traffic Control — third-framing Go implementation guide. Same three dials (server share + burst, per-query limit, max concurrent workers) applied to five tag axes. Canonical datum: the Go-side of the driver + middleware wire-up is where the five axes compose.
  • PlanetScale Insights — the extension that parses SQLCommenter tags + emits the SQLSTATE 53000 error / [PGINSIGHTS] notice. The post clarifies that both signals come from the same extension layer — Enforce as errors, Warn as notices — different wire- protocol frames.
  • PostgreSQL — substrate; NoticeResponse (Warn channel) + ErrorResponse SQLSTATE 53000 (Enforce channel) + application_name + username as driver-set tag axes independent of application-level tagging.
  • github.com/jackc/pgx/v5 — the canonical Go Postgres driver referenced by the post. pgx.ParseConfig + config.OnNotice
  • pgconn.PgError are the driver-specific integration points. (Not canonicalised as a new system page — narrow scope, no wiki-worthy architectural disclosure beyond the Traffic Control integration datum.)

Concepts extracted

  • concepts/sqlcommenter-query-tagging — the standard that underpins every tag axis. This post is the canonical Go- specific implementation of the standard (previously the standard was canonicalised via Rails implementations).
  • concepts/context-propagated-sql-tags — canonical new concept. Go's idiomatic alternative to ORM-middleware for threading per-request context through to SQL comment tags.
  • concepts/sqlstate-53000-traffic-control-error — canonical new concept. Enforce-mode error class with [PGINSIGHTS] Traffic Control: prefix, detected via errors.As on *pgconn.PgError.
  • concepts/postgres-notice-warning-channel — canonical new concept. Warn-mode observability delivered in-band via Postgres NoticeResponse frames, caught by the driver's OnNotice hook, orthogonal to SQL result rows.
  • concepts/composable-tag-axes — canonical new concept. Orthogonal tag dimensions AND-ed at enforcement time; the design principle that lets five separate patterns coexist on the same budget-rule engine.
  • concepts/warn-mode-vs-enforce-mode — already canonical from the graceful-degradation post; this post adds the Go-specific implementation of the Warn-mode telemetry- collection loop via OnNotice.
  • concepts/graceful-degradation — the user-facing outcome of the Enforce-mode shedding ("The difference between a database outage and a degraded experience...").
  • concepts/query-priority-classification — the prior post's canonical three-tier scheme; this post extends with non-priority axes (service / route / deployment / workload) that are orthogonal to but compose with priority.

Patterns extracted

Operational numbers

  • Retry-backoff budget: maxRetries = 3, initial 100ms, exponential doubling (100ms → 200ms → 400ms).
  • SetMaxOpenConns(4) for background-job pool — cap of 4 simultaneous connections as an application-layer concurrency floor orthogonal to Traffic Control's dial.
  • No adoption numbers, no production telemetry, no customer case study, no measured latency impact from tag rendering.
  • Pattern 7's Warn-mode observation window: "a few hours of representative traffic" (qualitative; no specific hour count).

Caveats

  • Pedagogical / how-to voice. Josh Brown is a developer- advocacy byline, not one of PlanetScale's canonical deep- internals authors. The post is genuinely architectural (five new patterns + two new mechanisms), but the voice is implementation-how-to rather than production-retrospective.
  • No Go-driver-specific benchmarking of tag-rendering cost. The appendTags call path runs on every query; for very high-QPS call sites, the sort.Strings + fmt.Sprintf + url.QueryEscape loop could add measurable overhead. Not quantified.
  • Plan-cache sensitivity to comment bytes not discussed deeply. sort.Strings gets deterministic ordering for the same tag set, but if a middleware adds / removes tags conditionally per-request, the tag set itself varies — still cache-miss-prone. Also depends on whether Postgres normalises over comments in pg_stat_statements (default: yes) or whether the extension in use preserves them.
  • URL-encoding choice. url.QueryEscape is used for values — works for keys like route='/api/export' containing / but means consumers reading the tags (e.g. SQL log analysers) must URL-decode. Not obvious from the raw SQL text.
  • Retry logic doesn't address thundering-herd risk. A spike that trips Enforce on every client simultaneously leads to synchronised retries — 3 × doubling = 100ms + 200ms + 400ms = 700ms window of retry traffic. No jitter in the canonical shape (operator is presumably expected to add it).
  • DEPLOYMENT_TAG pattern coverage is all-or-nothing. The env var is set at startup — all pods with the tag get the tag, pods without don't. A rolling deploy produces a mixed fleet where half the pods carry the tag. That's the point of the pattern (observe only new pods' behaviour) but implies the rule matcher must use equality, not prefix.
  • SaaS tier axis assumes a single-tenant subscription model. A workspace with multiple users at different tiers would complicate the tag assignment. Not addressed.
  • Background-jobs SetMaxOpenConns(4) datum is example- specific. The right number for a different application depends on job profile, database size, and concurrent-job cadence. Not derived.
  • No warn-mode logging volume discussion. If a budget is set too tight in Warn mode, every query emits a notice — log volume can explode. The post suggests aggregation ("log these notices to build an accurate picture") but doesn't specify the pipeline.
  • pgx/v5 specific. The OnNotice hook is a pgx feature; the standard database/sql interface does not expose notice handlers directly — other Go drivers would need their own integration path.

Source

Last updated · 378 distilled / 1,213 read