AWS ECS hosted omnibus gitlab cannot use AWS EC2 Elasticsearch IAM role instance credentials

Summary

Upgrading from Gitlab Omnibus 12.5.5 to 12.7.2 and from AWS ElasticSearch 6.5 to 6.8 (or 7.1) breaks integration with using IAM role instance credentials. As such, I suspect that the gitlab omnibus code has a bug in it. Currently the only work around is to use AWS IAM Access Keys instead (not preferred).

Steps to reproduce

We've had a gitlab omnibus installation since about version 10.4 and have migrated the application successfully up to 12.5.5 over the years. In attempting to move to 12.7.2, gitlab itself comes up fine, but I had tried a couple different migrations for the ElasticSearch integration. One attempt was to move from 6.5 to 6.8 (a simple AWS upgrade). Another attempt was to move from 6.5 to 7.1, which requires an entirely new ElasticSearch cluster. All infrastructure is created with Terraform.

What is the current bug behavior?

Initially, I was going to move to using ElasticSearch 7.1, and doing so needs a new cluster. So, I'd need to run two commands from within the docker container:

gitlab-rake gitlab:elastic:create_empty_index
gitlab-rake gitlab:elastic:index

The first command fails to create the empty index as shown below.

gitlab-rake gitlab:elastic:create_empty_index --trace
** Invoke gitlab:elastic:create_empty_index (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute gitlab:elastic:create_empty_index
rake aborted!
NoMethodError: undefined method `access_key_id' for #<Aws::InstanceProfileCredentials:0x00007f939af56740>
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/aws-sigv4-1.1.0/lib/aws-sigv4/signer.rb:670:in `credentials_set?'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/aws-sigv4-1.1.0/lib/aws-sigv4/signer.rb:661:in `get_credentials'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/aws-sigv4-1.1.0/lib/aws-sigv4/signer.rb:205:in `sign_request'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/faraday_middleware-aws-sigv4-0.3.0/lib/faraday_middleware/request/aws_sigv4.rb:22:in `sign!'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/faraday_middleware-aws-sigv4-0.3.0/lib/faraday_middleware/request/aws_sigv4.rb:14:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/faraday-0.15.4/lib/faraday/rack_builder.rb:143:in `build_response'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/faraday-0.15.4/lib/faraday/connection.rb:387:in `run_request'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/elasticsearch-transport-6.8.0/lib/elasticsearch/transport/transport/http/faraday.rb:23:in `block in perform_request'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/elasticsearch-transport-6.8.0/lib/elasticsearch/transport/transport/base.rb:266:in `perform_request'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/elasticsearch-transport-6.8.0/lib/elasticsearch/transport/transport/http/faraday.rb:20:in `perform_request'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/elasticsearch-transport-6.8.0/lib/elasticsearch/transport/client.rb:143:in `perform_request'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/elasticsearch-api-6.8.0/lib/elasticsearch/api/actions/info.rb:15:in `info'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/elastic/helper.rb:43:in `create_empty_index'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/tasks/gitlab/elastic.rake:60:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:273:in `block in execute'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:273:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:273:in `execute'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:214:in `block in invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/2.6.0/monitor.rb:235:in `mon_synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:194:in `invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:183:in `invoke'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:160:in `invoke_task'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:116:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:116:in `block in top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:125:in `run_with_threads'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:110:in `top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:83:in `block in run'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:186:in `standard_exception_handling'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:80:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/exe/rake:27:in `<top (required)>'
/opt/gitlab/embedded/bin/rake:23:in `load'
/opt/gitlab/embedded/bin/rake:23:in `<top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in `load'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in `kernel_load'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:28:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:463:in `exec'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor.rb:387:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:27:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/base.rb:466:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:18:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/exe/bundle:30:in `block in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/friendly_errors.rb:124:in `with_friendly_errors'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/exe/bundle:22:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:elastic:create_empty_index

What is the expected correct behavior?

gitlab-rake gitlab:elastic:create_empty_index
Index created

Relevant logs

Details of package version

gitlab-rake gitlab:env:info

System information
System:
Proxy:          no
Current User:   git
Using RVM:      no
Ruby Version:   2.6.5p114
Gem Version:    2.7.10
Bundler Version:1.17.3
Rake Version:   12.3.3
Redis Version:  5.0.7
Git Version:    2.24.1
Sidekiq Version:5.2.7
Go Version:     unknown

GitLab information
Version:        12.7.2-ee
Revision:       be1bc017799
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     PostgreSQL
DB Version:     10.9
URL:            <redacted>
HTTP Clone URL: <redacted>
SSH Clone URL:  <redacted>
Elasticsearch:  yes
Geo:            no
Using LDAP:     no
Using Omniauth: yes
Omniauth Providers: saml

GitLab Shell
Version:        11.0.0
Repository storage paths:
- default:      /var/opt/gitlab/git-data/repositories
GitLab Shell path:              /opt/gitlab/embedded/service/gitlab-shell
Git:            /opt/gitlab/embedded/bin/git

Environment details

Operating System: Amazon Linux AMI release 2018.03
Installation Target:
- VM: AWS
Installation Type, remove incorrect values:
- Upgrade from version 12.5.5
Is there any other software running on the machine: No
Is this a single or multiple node installation? No
Resources
- CPU: 4 VCPU
- Memory total: 16 GiB

Configuration details

Provide the relevant sections of `/etc/gitlab/gitlab.rb`