Skip to content

PATTERN Cited by 1 source

Time-bounded throttler rule

Pattern

Every operator-configured throttler rule — exemption, prioritisation, de-prioritisation, stricter threshold, routing override — carries a TTL. When the TTL expires, the rule is removed and behaviour reverts to the baseline (or the next-priority applicable rule).

The pattern is an operational-hygiene discipline: rules applied during an incident, a rush-hour window, or an investigation must not outlive the reason they were applied.

"Jobs and operations eventually complete. But it's also a good idea to put a time limit on any rules you may have set. If you've exempted a category of clients, then it's best if that exemption expires at some point. It is useful to de-prioritize all jobs for a couple of hours during rush hour if some unexpected workload is received, or for the duration of an ongoing investigation."

— Shlomi Noach, Source: sources/2026-04-21-planetscale-anatomy-of-a-throttler-part-3

Why time bounds

Throttler rules are adjusted during high-stress events: incidents, unexpected load spikes, investigations. The person configuring the rule is rarely the person who will reason about whether it's still appropriate three days later. Without a TTL:

  • Incident-time exemptions become permanent policy — the exemption added for the fix stays forever, silently removing the throttler's guarantee for that client.
  • Rush-hour suppressions outlast rush hour — the de-prioritisation applied Monday morning still applies Tuesday afternoon.
  • Investigation holds outlast the investigation — the block added to isolate a suspect client stays past resolution, starving that client indefinitely.

With a TTL the default failure mode is revert to baseline, not accumulate policy.

Mechanism

Each rule carries an explicit expiry:

rule_id:         "deprioritize-etls-incident-2026-04-21"
scope:           "*.etl.*"
type:            "rejection_ratio"
value:           0.90
created:         2026-04-21T10:05:00Z
expires:         2026-04-21T16:00:00Z
created_by:      "oncall@company"
reason:          "P1 incident INC-42 — database write path degraded"

At / past expires, the rule is removed from the active rule set.

Optional refinements:

  • Soft-warn before expiry (alert N minutes before so operators can explicitly extend or acknowledge).
  • Mandatory renewal justification for rules re-added with the same scope.
  • Default TTL per rule type (e.g. exemptions default to 1 hour; prioritisations default to 24 h). Unbounded rules should require an explicit flag or be disallowed.

Incident integration

A natural place for time-bounded rules is an incident- management workflow:

  • Incident opens → a rule set for this incident is active, tied to the incident ID.
  • Incident resolves → all rules scoped to that incident auto-expire (or alert if still in use).

This removes the "did we remember to revert throttler config?" post-incident checklist item.

Failure modes

  • Over-short TTLs in real incidents. A 10-minute exemption for a fix that actually takes 2 hours forces renewal churn. Default TTLs should match typical workload length.
  • Silent expiry of incident-critical rules. The operator who configured the rule may not be watching when it expires. Soft-warn before expiry helps.
  • TTL drift across system components. If the throttler and its consumers / observability layer use different clocks or caching, a "just expired" rule can linger in downstream caches.
  • Complex rule-stack interactions. A rule set with multiple overlapping TTLs can produce surprising baseline-restorations mid-operation.

What to put TTLs on

Rule type TTL default Rationale
concepts/throttler-exemption Short (minutes to hours) Riskiest shape; guaranteed starvation if stale.
Per-client rejection-ratio prioritisation Medium (hours) Operator judgement for the specific window.
Global de-prioritisation (rush-hour) Short (bounded by the known window) The window is by definition bounded.
Stricter thresholds for investigation Bounded by investigation completion Revert when investigation closes.
Baseline thresholds None These are standing policy, not adjustments.

Compositions

Seen in

Last updated · 319 distilled / 1,201 read