CONCEPT Cited by 1 source
Static sharding¶
Static sharding pins application keys to backend nodes using a fixed scheme — most commonly consistent hashing — computed by clients without a central coordinator. Simple, memory-local (each pod owns its keys and can cache state in-process), and fast on the hot path.
Three structural failure modes¶
Databricks' systems/dicer post enumerates them (Source: sources/2026-01-13-databricks-open-sourcing-dicer-auto-sharder):
- Unavailability during restarts and autoscale. No coordination with a cluster manager — the scheme reacts only after a pod is already gone, so rolling updates and scale events produce downtime or perf degradation.
- Prolonged concepts/split-brain and downtime during failures. Without central coordination, clients can develop inconsistent views of membership when pods crash or become intermittently unresponsive. Two pods may both believe they own the same key, or no pod may — dropping the customer's traffic.
- concepts/hot-key problem. The scheme cannot dynamically rebalance or replicate in response to load. A single hot key pins to one pod, which bottlenecks or cascades.
Why it still gets chosen¶
- Zero infrastructure — just a hash function.
- No control-plane dependency on the request path.
- Works fine at small scale / uniform load.
Contrast with dynamic sharding¶
- Static: assignment is a function of the key, fixed at code time.
- Dynamic (concepts/dynamic-sharding): assignment is state owned by a controller, updated in response to health + load signals.
At Databricks this trade-off forced teams into a painful binary: static-sharded-and-fragile vs concepts/stateless-compute-and-expensive. Dicer was built to make dynamic sharding the third option.
Seen in¶
- sources/2026-01-13-databricks-open-sourcing-dicer-auto-sharder — the three failure modes above are named as the motivation for Dicer.