Skip to content

SYSTEM Cited by 2 sources

Karpenter

Karpenter is an open-source (CNCF) Kubernetes node auto-scaler originally built by AWS. It watches pending pods, solves a bin-packing problem over configured instance types / zones / architectures, and dynamically provisions and de-provisions cloud instances directly — replacing the older Cluster Autoscaler model (which scales pre-defined node groups via ASGs) with per-instance control driven by actual pending pods.

Core design primitives

  • NodePool — declarative CRD that describes a pool of eligible nodes (limits, disruption policy, taints, labels, supported workloads).
  • EC2NodeClass — AWS-specific companion CRD describing the EC2-level details: allowed instance types, root volume size / IOPS / type / throughput, subnets, security groups, AMI family.
  • Bin-packing scheduler — picks the instance type + AZ + count that most efficiently absorbs the current set of pending pods.
  • Consolidation — continuously re-packs workloads onto fewer, larger nodes and retires the vacated ones; the inverse of scale-down-by-utilization-heuristic in CA.
  • Drift — detects when a running node's configuration has drifted from its NodePool / EC2NodeClass spec (e.g. AMI change) and replaces it.

Contrast with Cluster Autoscaler

The canonical wiki writeup is on the other page (systems/cluster-autoscaler). Summary:

Cluster Autoscaler Karpenter
Capacity primitive ASG (AWS) Direct EC2 RunInstances
Scaling latency Minutes Seconds
Instance diversity One template per ASG Many types per NodePool
AZ balance ASG-driven (poor) Scheduler-driven
Consolidation Utilization-heuristic Continuous bin-packing

What it solves

  • Flat over-provisioning of node pools that had to handle both steady-state load and deploy surges — expensive on nights and weekends.
  • Rigid node-group boundaries — Karpenter can span many instance types / zones / arch to match pod resource shapes in one NodePool.
  • Scale-down inefficiency of static fleets — continuous consolidation retires under-utilized nodes.
  • Subnet-pinned node pools — decoupling provisioning from specific subnets improves IP efficiency.
  • Poor AZ balance — the scheduler sees the whole cluster and picks under-represented AZs as it provisions.
  • Multi-minute scaling latency — pending-pod-driven provisioning collapses to seconds.

Pairs naturally with systems/keda or HPA for pod-level scaling: pods scale first, Karpenter scales nodes under them.

Known hazards (from production)

From Salesforce's 1,000-cluster migration (2026-01-12):

  • Bin-packing + consolidation can terminate singleton pods without warning. Mitigation: guaranteed-pod-lifetime features, workload-aware disruption policies, karpenter.sh/do-not-disrupt annotation.
  • PDB misconfigurations become migration-blocking. Overly restrictive or broken PDBs block node replacement. Fix upstream: audit + OPA-enforced PDB admission validation.
  • 63-character label limit can be unexpectedly breaking at scale because Karpenter's NodePool / EC2NodeClass matching is label-dependent — legacy human-friendly naming conventions often exceed the limit.
  • Ephemeral-storage defaults are not implicit. Moving from ASG to EC2NodeClass requires 1:1 volume-config translation; incomplete translation causes workloads to fail to schedule.
  • Parallel cordoning destabilizes clusters. Use sequential cordoning with verification checkpoints instead.

Scale references

  • Figma — scoped Karpenter into the ECS→EKS migration for cost savings; used with systems/keda for the pod layer (fast-follow).
  • Salesforce — 1,000+ EKS clusters / 1,180+ node pools / thousands of internal tenants. Canonical largest-known Karpenter production reference. Reports:
  • Scaling latency minutes → seconds.
  • 80% reduction in manual operational overhead.
  • 5% FY2026 cost savings, projected +5-10% FY2027.
  • Heterogeneous GPU / ARM / x86 in single node pools.
  • Eliminated thousands of node groups.
  • Datadog's State of Containers report (referenced in the 2026-01-12 post): +22% Karpenter-provisioned node share in the last 2 years across surveyed Kubernetes fleets.

Seen in

  • sources/2024-08-08-figma-migrated-onto-k8s-in-less-than-12-months — Figma scoped node-level auto-scaling (via Karpenter) into the ECS→EKS migration because cost savings justified the added scope for little extra work. Pod-level auto-scaling (Keda) was deferred to a fast-follow.
  • sources/2026-01-12-aws-salesforce-karpenter-migration-1000-eks-clusters — Salesforce's 1,000-cluster migration from Cluster Autoscaler + ASGs to Karpenter. Canonical wiki reference for Karpenter at extreme scale; documents the five operational lessons (PDB hygiene, sequential cordoning, 63-char labels, singleton protection, ephemeral-storage mapping), the three design principles of the in-house transition tool (zero-disruption + rollback + CI/CD-integrated), the automated ASG→Karpenter config mapping approach, and the rollout strategy (phased with soak times, risk-based sequencing).

Dual-taint strategy for compute-level tenant isolation

ReadyOn uses Karpenter as Wall 2 of a four-layer EKS tenant-isolation model. Karpenter provisioners create per-tenant nodes carrying two taints — a tenant-identifier taint (e.g. tenant=acme-corp) and a workload-type taint (e.g. workload=frontend) — and a pod must tolerate both to schedule. This makes Tenant A's frontend nodes distinct from Tenant A's batch nodes and both entirely separate from Tenant B's, so tenants never share compute nodes. A Kubernetes admission controller validates that a pod's toleration claims match its namespace's tenant identity, so even forged tolerations cannot force cross-tenant placement. Each node's IAM role is scoped to a single tenant and IMDSv2 runs with a reduced hop limit, so a container escape yields only single-tenant, node-level credentials — bounding blast radius. Platform nodes run on managed node groups with a separate taint no tenant workload can tolerate. This is a concrete production use of Karpenter's taints/NodePool primitives as a security boundary, not just a scheduling / bin-packing optimization. (Source: sources/2026-09-18-aws-readyons-four-walls-of-tenant-isolation-on-amazon-eks)

Last updated · 766 distilled / 2,225 read