GitLab Elasticsearch integration should only ever read/write from alias
Problem to solve
Today GitLab creates an index called gitlab-production and writes/reads this index in all places. Once the index is created with this name there is no way to rename it as indices cannot be renamed. Furthermore it is often recommended to use aliases when reading/writing to Elasticsearch as it gives you more flexibility to move/migrate indices transparently to the application.
The current design of GitLab makes it basically impossible even if an operator wants to use aliases because of the naming scheme of the index being hardcoded and index creation name being tied to index search/write name.
Intended users
Further details
This limited our options already when trying to reindex our own cluster. See:
- gitlab-com/runbooks!2017 (comment 305366295)
- gitlab-com/gl-infra/production#1902 (comment 318816298)
Some more thoughts at !17230 (comment 315195235)
Proposal
GitLab should always create an alias for reading/writing after creating the index. Furthermore the index name should be chosen to be something dynamic. Probably gitlab-production-<Time.now>. So when we do the initial index creation we create the index then the alias to put to that index. This should be backward compatible if the alias is now named gitlab-production since this will mean the even if the index was created before or after this feature the application is still reading/writing to gitlab-production.