Skip to content

ATLASSIAN 2026-07-24

Read original ↗

Online index migration and shard-scaling in OpenSearch with AOSC

Summary

Atlassian's Search Platform team (responsible for 300+ OpenSearch clusters, 2500+ data nodes, 1.7 PB of data across 13 regions) built AOSC (Atlassian OpenSearch Shard Control), an open-source OpenSearch plugin that performs zero-downtime index reshaping — changing shard counts, modifying mappings, or extracting noisy-neighbour tenants — without application-level dual-writes. AOSC applies database-style CDC techniques (bulk backfill via Lucene's Engine.Searcher + incremental replay via the Lucene changes snapshot + retention leases) to keep a target index in sync with the source, then atomically cuts traffic over using index aliases and a brief write-block.

Key Takeaways

  1. Existing OpenSearch primitives are inadequate for live reshaping. _reindex creates a stale Point-in-Time copy; _split/_shrink require a write-block for the entire operation; application dual-writes work but leak migration complexity into the indexer, require multi-week projects, and risk incidents.

  2. The CDC-from-database analogy translates directly. OpenSearch has no external CDC API, but retention leases (prevent segment deletion) + Lucene changes snapshot (operation-level changelog per shard) provide equivalent primitives. Combined with Engine.Searcher for bulk backfill, you get backfill → replay → converge → cut over.

  3. Coordination splits into cluster-manager and data-node roles. An AoscCoordinatorService on the cluster-manager node orchestrates index-level lifecycle (tracking convergence, write-blocking, alias swap). AoscShardService on each data node spawns per-shard ShardMigrationWorkers that do the local backfill/replay work. Communication uses cluster state (coordinator → shards) and transport actions (shards → coordinator).

  4. The cutover window uses a brief write-block, not held writes. Once all shards converge within a 500-op gap, the coordinator write-blocks the source, does one final replay per shard, validates doc counts, and swaps the alias. Applications must classify write-block errors as retryable.

  5. Overload protection is the key production concern. On a live cluster, unthrottled backfill saturates CPU and hurts search latency. AOSC solves this with: transient target settings (0 replicas, no refresh during backfill), max 1 concurrent source shard per node, configurable convergence thresholds, exponential backoff on overload, and pluggable rate controllers (fixed, adaptive batch via AIMD, or Gradient2-tuned adaptive).

  6. Custom routing keys introduce a replay correctness challenge. Translog.Delete doesn't carry routing keys. For same-shard-count migrations, AOSC synthesizes a routing key. For power-of-2 splits, it fans deletes across the target shard block. Non-power-of-2 changes remain unsupported pending upstream work.

  7. Scale numbers: 300+ clusters, 2500+ data nodes, 1.7 PB data, 13 regions. The ~150-person Search Platform org serves Quick Search, Full Page Search, Rovo Search, Smart Answers, and Rovo Chat.

Architecture (simplified)

Start migration
→ For each shard:
    → Acquire retention lease
    → Backfill via Engine.Searcher
    → While gap > 500:
        → Replay Lucene changes snapshot
        → Advance retention lease
→ Write-block source index
→ For each shard: final replay
→ Verify document count (source == target)
→ Alias swap (atomic)
→ Complete migration

Operational Numbers

Metric Value
Clusters managed 300+
Data nodes 2500+
Total data 1.7 PB
Regions 13
Search Platform team size ~150 people
Default concurrent source shards per node (prod) 1
Convergence threshold (ops gap) 500

Caveats

  • Non-power-of-2 shard count changes with custom routing keys are unsupported (pending OpenSearch#20907).
  • Applications must already retry write errors for the brief cutover window to work (Search Platform had this in place).
  • AOSC is designed for OpenSearch; not portable to Elasticsearch without adaptation.
  • The brief write-block means a short window of stale search results is still possible (though much shorter than full _split/_shrink).

Source

Last updated · 596 distilled / 1,814 read