---
title: Online index migration and shard-scaling in OpenSearch with AOSC
source: Atlassian Engineering
source_slug: atlassian
url: https://www.atlassian.com/blog/how-we-build/opensearch-aosc-plugin
published: 2026-07-24
fetched: 2026-07-28T14:16:38+00:00
ingested: true
---

**Learn how Atlassian built[AOSC](https://github.com/atlassian-labs/opensearch-aosc), an OpenSearch plugin to reshape indices and scale their shards online.**

**AOSC** is a **zero-downtime*** index migration & scaling engine for [OpenSearch](https://docs.opensearch.org/latest/getting-started/intro/), built as an OpenSearch plugin. It’s **open source** ([GitHub repo](https://github.com/atlassian-labs/opensearch-aosc), [docs](https://atlassian-labs.github.io/opensearch-aosc/)), and this post is the story of how, and why, we built it. Anyone who has worked with OpenSearch would have, at some point, run into one of the following problems:

  1. I provisioned the index with N shards, and now my shards are large. I want to scale up to 2N shards. _How can I change the number of shards?_
  2. I created my index with a `text` type mapping, but now I want to change it to `keyword`. _How can I change the mappings?_
  3. I created this mapping field which I thought would be useful, but I don’t search on it anymore. _How can I delete fields from the mappings?_



As the team behind the OpenSearch clusters that power Atlassian Search & Rovo, we run into these questions **on a daily basis**. We’re constantly working on projects to improve search performance for our customers, and on projects like index-per-tenant to isolate customers with large amounts of data. We needed a way to solve them efficiently: minimal cost, minimal interruption, minimal extra code, and no stress on our clusters.

## About the team

We’re the Core Index Serving team in **Search Platform** at Atlassian, the ~150-person org behind a lot of search surface in the company: Quick Search, Full Page Search, [Rovo](https://www.atlassian.com/software/rovo) Search, Smart Answers, Rovo Chat, and the AI experiences; in products like Confluence and Jira, all of which need search to be fresh and correct. Our team owns the OpenSearch clusters underneath all of it: **300+ clusters** , **2500+ data nodes** , and **1.7PB of data** (constantly growing in this AI world), a mix of shared-tenant and per-tenant clusters across **13 regions**.

## OpenSearch in one picture

Everything AOSC reshapes lives in a handful of pieces. It helps to have them in mind before the rest of the post:

![](https://atlassianblog.wpengine.com/wp-content/uploads/2026/07/opensearch-building-blocks-20260725-045240.png)

## Challenges we face with scale

At the scale of Search Platform, changing the shape of an index isn’t a rare event; it’s a constant, and the demand only grows. More products and third-party sources land on the same clusters, more engineers build on them, and the pace of iteration keeps rising. As the product and its search relevance evolve, the indices underneath have to evolve with them, and each of those changes lands on a live index that’s serving user searches the whole time.

So _“can you reshape this index?”_ lands in our queue constantly, in shapes like these:

  * **Large tenants in shared indices becoming a noisy neighbour** for other tenants – We want to extract such tenants into their own dedicated indices.
  * **Tenant growth** – When a tenant grows on a dedicated index, we want to increase their number of shards.
  * **Removing fields to improve performance** – We’ve noticed significant performance gain by dropping all vector embeddings from indices which don’t require it.
  * **Changing a field type** from `text` to `keyword`.
  * **Flattening a nested field** in the index mapping.



For a lean team, doing each of these by hand, teaching the indexer to dual-write, babysitting the backfill, and choreographing the cutover, stops being a task and becomes a **tax**. What we wanted was for OpenSearch to just _do the migration itself_ , safely, on live traffic.

Longer term, safely reshaping a live index is also the groundwork for **auto-scaling shards** : a policy that grows an index’s shards as it outgrows them, no human in the loop. We still trigger it by hand today, but the hard part, _reshaping a live index without downtime_ , is the same whether a human or a policy presses the button.

## Options that existed

So what does OpenSearch actually give you for this today? We reached for each of these in turn, and on a live index none of them quite works, for different reasons.

Option| What it does| The catch on a live index  
---|---|---  
[/_reindex](https://docs.opensearch.org/latest/api-reference/document-apis/reindex/)| A Point-in-Time (PIT) copy of the source index into a new target index| It doesn’t see writes that land after the copy starts, so the _target is stale_ the moment it finishes  
[/_split](https://docs.opensearch.org/latest/api-reference/index-apis/split/), [/_shrink](https://docs.opensearch.org/latest/api-reference/index-apis/shrink-index/)| Change the shard count in place| Both require a **write-block** on the source for the whole operation, so search goes stale while it runs  
**Application dual writes**|  The app writes to both old and new, bootstraps the target from its source data, verifies the two are _equal_ , then cuts reads over| The only option that keeps the index online, but it pushes the entire migration into your application (more below)  
  
### Why the options are not sufficient

The first two don’t work for us because we want to keep our indices online and available for writes. A write-blocked index means **stale search results** for users. That pushes teams toward dual writes, which is where the real cost shows up:

  * Setting up dual writes requires coding, testing, and cleanup. **Migration logic leaks into the indexer’s code.** It suddenly needs rollout choreography, dual-write behavior, retry handling, reconciliation, and cleanup, including awkward cases like _“the write succeeded on one index but failed on the other due to a version conflict, now what?”_. That’s a lot of custom correctness logic to bake into a service whose real job is just to index documents.
  * Determining whether the source and target indexes are equal is notoriously hard. You’re **never quite sure** the target has fully caught up, and you usually end up accepting a _~0.1% doc-count difference_ as “close enough”.
  * Bootstraps from the source are **expensive** , burning read and compute across multiple layers of services, and that operational traffic has to compete with live indexing events.
  * Shard count and index shape are **search-infrastructure concerns, not application concerns** , yet dual writes push that operational burden onto the indexer and the team that owns it.



Overall, what should be a single API call becomes a **multi-week project** with real potential to **cause incidents**.

![](https://atlassianblog.wpengine.com/wp-content/uploads/2026/07/before-after-20260725-045233-scaled.png)

## The missing gap

None of this is unique to search, though. Changing a live system’s shape without downtime is a well-worn problem in the database world, so that’s where we looked first. Imagine you had to change a column from `int` to `bigint` on a busy SQL database without taking the application down, and often enough that you’d build a tool for it. How would you do that? For me, the approach would be: an initial bulk copy from source to target, then a **CDC pipeline** replaying every change that lands afterward, and once the gap is _small enough_ , a short cutover, an atomic swap of table names.

### OpenSearch investigation

We set out to do the same research for OpenSearch. It turns out OpenSearch has no externally exposed CDC API, but it does have two building blocks: **Retention Leases** and the **Lucene Changes Snapshot**. Together, these give us the equivalent of CDC.

The next question was: _how do we do the one-time backfill from source to target before replaying the changes snapshot?_ There are a few options; the one we chose is an `Engine.Searcher`. It pins the relevant Lucene segments (marks them _“don’t delete while pinned”_) so the data stays searchable for the scan. It doesn’t stop merges: merges still happen and new segments still form.

The last question: _how do we cut over from source to target atomically, and make sure the application ends up pointing at the target?_ The answer is [**index aliases**](https://docs.opensearch.org/latest/im-plugin/index-alias/): a stable name that points at an index and can be flipped to another one atomically. So we create the target, bootstrap it, replay the changes snapshot, and swap the alias. And **luckily** , Search Platform has always talked to its OpenSearch clusters through aliases.

### The last mile: cutover

There’s a small window between the last replay and the alias swap. _What happens to a write that lands on the source index during that gap? How do we make sure those tail events reach the new index too?_ We explored two options:

  1. **A brief write block** on the source index once the gap is minimal – applications see (retryable) write errors for a moment.
  2. **Holding writes at the alias** during the swap, then releasing them – applications see a short latency bump instead.



We chose the **write block** , since it’s a native OpenSearch concept. All it needs is for the application layer to be resilient, classify these as **retryable errors** , and retry them. **Luckily** , that too was already in place for us. Our indexing application is resilient to such failures and does thorough retries.or the application layer to be resilient, classify these as **retryable errors** , and retry them. **Luckily** , that too was already in place for us. Our indexing application is resilient to such failures and do thorough retries.

## Architecture

Here’s the whole migration at a 10,000-foot view:
    
    
    → Start migration
    → For each shard {
      → Acquire retention lease
      → Backfill using Engine.Searcher
      → while gap is more than 500 {
        → Replay lucene changes snapshot
        → Advance retention lease
      }
    }
    → Write block source index
    → For each shard { → Do one final replay }
    → Verify document count between source and target
    → Alias swap
    → Complete migration

The shape is deliberately simple. But safely orchestrating this whole migration, and getting confidence by thorough testing takes a lot of time. Below we’ll go into some of those details, and challenges we faced while developing it.

![](https://atlassianblog.wpengine.com/wp-content/uploads/2026/07/lifecycle-20260725-045234-scaled.png)

## Deep dive

### State management and communication

An important design decision is _where_ the work runs. Every primitive AOSC depends on, the **Engine.Searcher** , the **changes snapshot** , and **retention leases** , is **shard-local** : it’s only available on the node that holds that shard, through its **IndexShard**. Streaming all of that data out to an external process would be slow and fragile. So AOSC runs the work _inside_ the cluster, right next to the shards, and splits it into two roles:

  * **On every data node** , an **AoscShardService** watches for active migrations and, for each source primary shard it hosts, spawns a **ShardMigrationWorker**. The worker does the shard-local work: acquire the retention lease, backfill via the Engine.Searcher, replay the changes snapshot, converge, and report progress.
  * **On the cluster-manager node** , an **AoscCoordinatorService** runs a **MigrationCoordinator** per migration. It owns the index-level orchestration: tracking global progress, deciding when _every_ shard has converged, applying the write block, validating, and performing the cutover.



The two halves talk through two channels:

**Coordinator → Shards.** The coordinator records the migration and per-shard phase in [**cluster state**](https://docs.opensearch.org/latest/api-reference/cluster-api/cluster-state/) (OpenSearch’s versioned, cluster-wide metadata, replicated to every node); data nodes listen for those changes and advance the right workers. When it’s time for the final post-write-block replay, that signal travels through cluster state too.

**Shards → Coordinator.** Shard workers report periodic progress and push important transitions immediately, for example _“this shard is now within 500 ops of the source”_ , via a **transport action** straight to the coordinator.

![](https://atlassianblog.wpengine.com/wp-content/uploads/2026/07/coordination.png)

To see exactly how the coordinator and shards interact, there’s an interactive walkthrough at [How AOSC Works | AOSC](https://atlassian-labs.github.io/opensearch-aosc/develop/how-it-works) .

### Monitoring

A migration you can’t watch is a migration you can’t trust, especially one that runs for hours against a live cluster. AOSC exposes the full state of a migration through a status API, the phase of every shard, each shard’s replay gap and convergence, throughput, and where the long tail is, so you can poll it and build dashboards like the one below.

![Monitoring dashboard](https://atlassianblog.wpengine.com/wp-content/uploads/2026/07/monitoring-dashboard-1.png)

### Custom routing key: a special case

This one only matters if you use custom routing keys, as we do to solve multi-tenancy, so treat it as a special case rather than core to the design. The catch is in replay: the lucene changes snapshot hands _deletes_ back **without** the original routing key (`Translog.Delete` doesn’t carry the routing key), so a naively replayed delete can route by `_id` to the wrong shard and _miss the document entirely_.

How AOSC handles it depends on how the shard count is changing:

  * **Same shard count** – synthesize a routing key that lands the delete on the correct target shard.
  * **Power-of-2 split** – a split maps each source shard onto a fixed, contiguous block of target shards, so we fan the delete out across that whole block (a harmless no-op on the shards that don’t hold the document). AOSC rejects the migration up front unless the source and target share the same `number_of_routing_shards`.
  * **Anything else** (shrink, or a non-power-of-2 change) – there is no safe, routing-free delete today, pending [OpenSearch #20907](https://github.com/opensearch-project/OpenSearch/issues/20907).

![](https://atlassianblog.wpengine.com/wp-content/uploads/2026/07/routing-split-20260725-045238-scaled.png)

The full routing math (the MurmurHash3 routing-shard space, the fan-out proof, and the synthetic-key mechanism) lives in [Routing and replay](https://atlassian-labs.github.io/opensearch-aosc/develop/concepts/routing-and-replay).

### Preventing cluster overload

For a live cluster serving searches, this is the part that matters most: _how do you reshape an index without degrading search for everyone else on the cluster?_ Our first staging runs made the problem visible. On an idle cluster, AOSC finishes migrations very quickly and drives high CPU on the target nodes, which is fine offline, but on a cluster serving live user searches, that same intensity would hurt reliability and latency. (The flip side: for an offline migration, AOSC doubles as a faster, more resilient /_reindex alternative.)

So the design goal flipped from _fast_ to _deliberately slow, steady, and tunable to whatever headroom you have_. The levers you get:

  1. **Transient target settings** – We apply throughput-friendly settings to the target during backfill (by default `{"index.number_of_replicas":"0","index.refresh_interval":"-1"}`), configurable as a cluster default and per migration, and reverted once shards reach minimal convergence. This applies only to the backfill phase, not replay.
  2. **Maximum concurrent source shards per node during backfill** – The main throttle to prevent overload. We run **1** in our production clusters, which keeps the migration slow, steady, and very low on CPU overhead.
  3. **Convergence threshold and max rounds** – These tune “when a shard tells the coordinator it has converged” (the coordinator only write-blocks once _all_ shards have). If a shard can’t reach the threshold within N rounds, it marks itself failed, producing a clean failure rather than a stuck migration.
  4. **Overload protection** – AOSC rides out indexing backpressure, admission control, and other transient errors with exponential backoff and jitter, only failing the migration after `aosc.overload.max_consecutive_failures`.
  5. **Pluggable rate controllers** via `aosc.<backfill|replay>.controller.*`. The default is `adaptive_batch` (the _safe choice_ for a live cluster), which is also what we run in production. Three controllers ship: **Fixed** (fixed batch size and concurrency), **Adaptive batch** (AIMD on batch size, reacting to backpressure and admission-control rejections), and **Adaptive** (Gradient2 tuning of both batch size and concurrency).



Full settings reference: [AOSC configuration](https://atlassian-labs.github.io/opensearch-aosc/develop/reference/configuration).

## Why we open-sourced it

This problem is not unique to Atlassian. Anyone operating large OpenSearch clusters eventually runs into the gap between _“I can copy this index”_ and _“I can safely move live traffic to a changed index.”_ Open-sourcing it forced useful discipline: the quickstart has to work outside our environment, the limitations have to be written down, and the failure model has to be explainable to someone who was not in the design room.

_That pressure made the project better._

## Try it

The fastest way to understand AOSC is to run a small local migration:

## The real lesson

Online schema change sounds like plumbing. In practice, it is a **coordination problem hiding inside a routine maintenance task**.

You have live writes. You have old data. You have a target index that must become correct before traffic moves. You have a short cutover window where the system needs to stop accepting writes, catch up, validate, and make one careful switch.

AOSC gives that problem a better home: _inside OpenSearch, close to shards, operation history, cluster state, and aliases._

The result is not magic. It is a controlled workflow:

**Backfill. Replay. Converge. Block briefly. Catch up. Validate. Cut over.**

_That is the part worth building._

## References

**AOSC**

**OpenSearch**

_Built with heart using_[ _Rovo Dev_](https://www.atlassian.com/software/rovo-dev) _, Atlassian’s AI coding agent._