PATTERN Cited by 1 source

Historical-usage auto-rightsizing¶

Pattern: close the feedback loop between a running quota-enforcing system and its own provisioning by (a) capturing enforcement + usage statistics as data-plane telemetry, (b) persisting them in a queryable lakehouse / warehouse substrate, and (c) running a separate rightsizing service that consumes that history on a cadence, applies forecasting strategies (organic growth, bursts, underutilization), and writes new quota values back through the control-plane API.

Shape¶

┌──────────────────┐                ┌────────────────────┐
│  Data plane      │                │  Control plane     │
│  (enforcers)     │                │  (quota lifecycle) │
│                  │                │                    │
│  emits usage     │                │  serves quotas     │
│  statistics      │                │  to data plane     │
└───────┬──────────┘                └────────▲───────────┘
        │                                    │
        │                                    │ write new
        │                                    │ quota values
        ▼                                    │
┌──────────────────────────────────┐   ┌────┴────────────┐
│  Pre-aggregated telemetry store  │   │  Rightsizing    │
│  (Iceberg on S3 / similar)       │──▶│  service        │
└──────────────────────────────────┘   │                 │
                ▲                      │  strategies:    │
                │                      │  - org growth   │
                │                      │  - bursts       │
                │                      │  - underuse     │
                │                      └─────────────────┘
                │                                │
                │                                └── reads from
                │                                    Iceberg /
                │                                    Presto /
                │                                    user sources
                │
       (pre-aggregation keeps
        storage bounded)

Pinterest's implementation¶

Per the Piqama post (Source: sources/2026-02-24-pinterest-piqama-pinterest-quota-management-ecosystem):

Telemetry. Piqama clients transparently collect enforcement + usage statistics. For applications not using Piqama clients, system- or storage-based feedback loops exist.
Pre-aggregated persistence. Statistics land in Apache Iceberg on S3 with a predefined schema. Pre-aggregation at write time is explicit: "These stored statistics are also pre-aggregated to optimize storage space."
Separate rightsizing service. "Piqama's framework allows a separate auto-rightsizing service to continuously consume historical data from various sources, including Presto, Iceberg, and user-defined data sources." The service is decoupled from the Piqama control plane; it operates as a consumer + API-writer.
Forecasting strategies. "This service applies rightsizing strategies designed to predict needs based on organic usage growth, traffic bursts, and underutilization detection."
Write-back. The new quota value flows through Piqama's normal CRUD + validation + dispatch, so rightsizing-generated updates respect the same ownership + invariants as operator-generated updates.

Design properties¶

Decoupled from the hot path. Rightsizing is an offline / batch workload; it cannot affect data-plane latency.
Decoupled from the control plane. The service runs as an independent process + API client; outage of rightsizing doesn't break quota enforcement. Worst case: quotas drift from optimal.
Pluggable data sources. Per above, rightsizing reads from Iceberg / Presto / user-defined sources — domains with their own telemetry can integrate without touching the store.
Manual override mandatory. Operators retain a manual-adjust path for firefighting; see quota auto-rightsizing caveats.

Why Iceberg + Presto¶

The substrate choice matters: Iceberg on S3 gives cheap + durable storage with standard analytics access (Presto / Spark / Trino / Athena / Flink). An alternative — a bespoke time-series database — would require dedicated capacity, bespoke querying, and would likely fan-out into per-application telemetry stores. The lakehouse choice reuses the org's existing analytics infrastructure for an additional consumer.

Trade-offs¶

Latency of feedback. Iceberg + batch rightsizing means the loop runs on an hours-to-days cadence; not useful for sub-minute adaptive capacity. Real-time demand adjustment still needs autoscaling.
Forecasting risk. Underestimating bursts can throttle legitimate traffic; overestimating wastes resources. Strategies need per-domain tuning and should include a safety factor.
Coverage gap. Applications not emitting usage telemetry are invisible to rightsizing. Piqama's "system-based and storage-based feedback loops" for non-client integrations is a mitigation but requires extra engineering per integration.

Seen in¶

sources/2026-02-24-pinterest-piqama-pinterest-quota-management-ecosystem — canonical wiki instance: Piqama's rightsizing service + Iceberg on S3 + Presto, deployed for capacity-based quotas on the Big Data Processing Platform. Rate-limit rightsizing is listed as "done centrally via Piqama right-sizing service, by periodically aggregating the request usage stats, forming the feedback loop."