elasticsearch omnibus e2e test setup broken - elastic:create_empty_index rake task fails with NoMethodError

Elasticsearch Integration Failure: NoMethodError: undefined method 'merge' for nil in Elasticsearch Client

Description

The Elasticsearch E2E test is consistently failing with a NoMethodError when attempting to run rake task elastic:create_empty_index

https://gitlab.com/gitlab-org/gitlab/-/jobs/9983025482

Note: also affects search-related nightly jobs:

• integration-elasticsearch-compatibility-version-7 1/1
• integration-elasticsearch-compatibility-version-8 1/1
• integration-opensearch-compatibility-version-1 1/1
• https://gitlab.com/gitlab-org/gitlab/-/jobs/10157051349

Error Details

NoMethodError: undefined method `merge' for nil
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gem_extensions/elasticsearch/model/client.rb:28:in `block in client'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gem_extensions/elasticsearch/model/client.rb:23:in `synchronize'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gem_extensions/elasticsearch/model/client.rb:23:in `client'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/elastic/helper.rb:33:in `initialize'

The job retried this operation three times with the same error, indicating this is a persistent issue with the configuration rather than a transient failure.

Environment

• GitLab EE: registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:9efaccb5b5028cd59e08bf94592f7c980ea80265-ruby3.3.7
• Elasticsearch: docker.elastic.co/elasticsearch/elasticsearch:8.2.0

Related Links

• Elasticsearch Integration Documentation
• CI Job: elasticsearch 1/1 (#9983025482)

Investigation

By running gitlab-qa against 17.11 vs 18.0 it looks like this was probably introduced in %18.0

$ bundle exec exe/gitlab-qa  Test::Integration::Elasticsearch gitlab/gitlab-ee:17.11.0-ee.0 --no-tests
$ bundle exec exe/gitlab-qa  Test::Integration::Elasticsearch gitlab/gitlab-ee:18.0.0-ee.0 --no-tests