## Problem As GitLab scales, so do the size of individual shards within Advanced Search indexes and the indexes themselves. Currently, our maintenance of this problem includes: 1. Create new index with more primary shards 2. Pause indexing 3. Reindex data from old index to new index 4. Point alias to new index 5. Resume indexing The manual intervention does not scale **AND** it does not help self managed instances with shard size imbalance. ## Proposal Create a scalable automated solution to handle shards becoming too big. This will keep all of our indexes around the same size with the same number of shards which would make capacity planning more predictable as the system scales. We can create a custom worker or utilize [Index Lifecycle Management Policy (ILM policy)](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-lifecycle-management.html) for segments based on `namespace_id`. This will include: * Create worker/policies using Advanced Search Migrations to push changes out the self managed users * Use read and write aliases * We'll have a fixed number of aliases for every document type * Example * index name: projects-1-0001, projects-2-0002 * Alias: projects-1 (all indices for read and the last one for write) * Global search: projects-* * Group search: <namespace_id> % <number_of_aliases> = <index_alias_name> ```mermaid graph TD R{Sharded Read} --> A1(alias: issues-1) R --> A2(alias: issues-2) A1 --> |ILM| I1(index: issues-1-1) A1 --> |ILM| I2(index: issues-1-2) A2 --> |ILM| I3(index: issues-2-1) A2 --> |ILM| I4(index: issues-2-2) GR(alias: issues) --> I1 GR --> I2 GR --> I3 GR --> I4 RR{Sharded Write} --> AA1(alias: issues-1) RR --> AA2(alias: issues-2) AA1 -.-> II1(index: issues-1-1) AA1 --> II2(index: issues-1-2) AA2 -.-> II3(index: issues-2-1) AA2 --> II4(index: issues-2-2) ```