Advanced Search: Index lifecycle management [Phase 1]

This issue is part of Index lifecycle management for Advanced Search indices

As our first step my proposal is to pretend that we have number_of_aliases set to 1 and then:

  1. Ship a migration to set is_write_index for existing aliases
  2. Ensure that when we index something we delete documents we're indexing from read-only indices (bulk indexer & elasticsearch-indexer. Possibly _bulk delete requests with the same document ids)
  3. Create an ILM Sidekiq worker (in my opinion it will be better than using Index lifecycle policies) that would be able to create and link new indices to existing aliases based on our own policy (shard size/document count/age?)
  4. Adapt our migration framework to supporting multiple indices per type
  5. Enable the ILM worker on staging and then on GitLab.com behind a feature flag
  6. Enable the worker by default
Edited by Dmitry Gruzd