SYSTEM Cited by 1 source
Moka (Pinterest Big Data Processing Platform)¶
Moka is Pinterest's next-generation massive-scale Big Data Processing platform for batch jobs, built atop the Apache open-source Yunikorn resource-scheduling framework. Yunikorn manages memory / GPU / CPU for batch-processing jobs; Moka is the higher-level platform surface exposed to data-engineering teams at Pinterest.
Role in this wiki¶
Canonical wiki entry point for Moka as a named Piqama integration target — specifically the capacity-quota variant, where Piqama manages physical-resource quotas (memory, vcore, max-concurrent-apps) per project via Yunikorn queues.
Quota mechanics (per Piqama post)¶
- Each project → one Yunikorn queue.
- Piqama stores three per-project quota values:
- Guaranteed Resources — minimum memory + vcore the project is guaranteed to receive.
- Maximum Resources — upper limit of memory + vcore.
- Max Concurrent Applications — simultaneous running-app cap.
- A Yunikorn Config Updater regularly polls Piqama and rewrites Yunikorn configs accordingly.
- On application completion, Yunikorn's Application Summary (including resource usage) lands in an S3 file. An aggregation job builds a resource database that drives both (a) future quota auto-rightsizing and (b) budget-enforced throttling.
Quota values are generated two ways: Auto Rightsizing (sliding- window historical usage; Moka's "legacy" mode) and Manual Adjustments for firefighting. The team is developing a budget- based approach for future quota generation.
Budget enforcement¶
When a project exceeds its allocated budget inside a defined time window, Piqama dynamically lowers that project's Maximum Resources. The haircut is tier-weighted (X% depending on the project's tier) — a deliberate graceful degradation rather than a binary cutoff. Teams must either secure additional budget or re-prioritise workloads if they're not critical.
Canonical wiki instance of budget-enforced quota throttling paired with chargeback cost attribution.
Seen in¶
- sources/2026-02-24-pinterest-piqama-pinterest-quota-management-ecosystem — canonical wiki introduction. Moka is the capacity-quota integration worked example; Yunikorn Config Updater is the bridge between Piqama's control plane and Yunikorn's scheduler.
Caveats¶
This stub page captures Moka only as described in the Piqama post. A dedicated Moka post is linked in the source but not yet ingested; Moka's own architecture — Yunikorn usage at scale, Kubernetes deploy model, storage substrate, job-scheduling policies — is not covered here and awaits the Moka-specific ingest.