Rake task gitlab:elastic:index errors with Index or alias under 'gitlab-production' already exists

Summary

Active indexing jobs could create the main gitlab-production alias and underlying index if data is added while this rake task is run. This is possible on very busy self managed instances or if a large project is in the middle of indexing while this task is run.

This can happen when users are running gitlab:elastic:recreate or gitlab:elastic:index rake tasks.

Discovered while debugging a self managed instance: https://gitlab.com/gitlab-com/ops-sub-department/section-ops-request-for-help/-/issues/155#note_1480905994

Steps to reproduce

1. make sure indexing is enabled
2. run the first step from the gitlab:elastic:recreate rake task: gitlab:elastic:delete_index
3. push any commit or repository file change
4. run the second step from the gitlab:elastic:recreate rake task: gitlab:elastic:create_empty_index
5. get an error

What is the current bug behavior?

Index creation fails

What is the expected correct behavior?

Index creation succeed

Relevant logs and/or screenshots

➜ bundle exec rake gitlab:elastic:create_empty_index
rake aborted!
Index or alias under 'gitlab-development' already exists.
/Users/terrichu/projects/gitlab-development-kit/gitlab/ee/lib/gitlab/elastic/helper.rb:424:in `create_index'
/Users/terrichu/projects/gitlab-development-kit/gitlab/ee/lib/gitlab/elastic/helper.rb:204:in `create_empty_index'
/Users/terrichu/projects/gitlab-development-kit/gitlab/ee/lib/tasks/gitlab/elastic.rake:165:in `block (3 levels) in <main>'
/Users/terrichu/.asdf/installs/ruby/3.1.4/bin/bundle:25:in `load'
/Users/terrichu/.asdf/installs/ruby/3.1.4/bin/bundle:25:in `<main>'
Tasks: TOP => gitlab:elastic:create_empty_index
(See full trace by running task with --trace)

Possible fixes

• ee/lib/tasks/gitlab/elastic.rake