AWS ECS hosted omnibus gitlab cannot use AWS EC2 Elasticsearch IAM role instance credentials
Summary
Upgrading from Gitlab Omnibus 12.5.5 to 12.7.2 and from AWS ElasticSearch 6.5 to 6.8 (or 7.1) breaks integration with using IAM role instance credentials. As such, I suspect that the gitlab omnibus code has a bug in it. Currently the only work around is to use AWS IAM Access Keys instead (not preferred).
Steps to reproduce
We've had a gitlab omnibus installation since about version 10.4 and have migrated the application successfully up to 12.5.5 over the years. In attempting to move to 12.7.2, gitlab itself comes up fine, but I had tried a couple different migrations for the ElasticSearch integration. One attempt was to move from 6.5 to 6.8 (a simple AWS upgrade). Another attempt was to move from 6.5 to 7.1, which requires an entirely new ElasticSearch cluster. All infrastructure is created with Terraform.
What is the current bug behavior?
Initially, I was going to move to using ElasticSearch 7.1, and doing so needs a new cluster. So, I'd need to run two commands from within the docker container:
gitlab-rake gitlab:elastic:create_empty_index
gitlab-rake gitlab:elastic:index
The first command fails to create the empty index as shown below.
gitlab-rake gitlab:elastic:create_empty_index --trace
** Invoke gitlab:elastic:create_empty_index (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute gitlab:elastic:create_empty_index
rake aborted!
NoMethodError: undefined method `access_key_id' for #<Aws::InstanceProfileCredentials:0x00007f939af56740>
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/aws-sigv4-1.1.0/lib/aws-sigv4/signer.rb:670:in `credentials_set?'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/aws-sigv4-1.1.0/lib/aws-sigv4/signer.rb:661:in `get_credentials'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/aws-sigv4-1.1.0/lib/aws-sigv4/signer.rb:205:in `sign_request'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/faraday_middleware-aws-sigv4-0.3.0/lib/faraday_middleware/request/aws_sigv4.rb:22:in `sign!'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/faraday_middleware-aws-sigv4-0.3.0/lib/faraday_middleware/request/aws_sigv4.rb:14:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/faraday-0.15.4/lib/faraday/rack_builder.rb:143:in `build_response'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/faraday-0.15.4/lib/faraday/connection.rb:387:in `run_request'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/elasticsearch-transport-6.8.0/lib/elasticsearch/transport/transport/http/faraday.rb:23:in `block in perform_request'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/elasticsearch-transport-6.8.0/lib/elasticsearch/transport/transport/base.rb:266:in `perform_request'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/elasticsearch-transport-6.8.0/lib/elasticsearch/transport/transport/http/faraday.rb:20:in `perform_request'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/elasticsearch-transport-6.8.0/lib/elasticsearch/transport/client.rb:143:in `perform_request'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/elasticsearch-api-6.8.0/lib/elasticsearch/api/actions/info.rb:15:in `info'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/elastic/helper.rb:43:in `create_empty_index'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/tasks/gitlab/elastic.rake:60:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:273:in `block in execute'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:273:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:273:in `execute'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:214:in `block in invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/2.6.0/monitor.rb:235:in `mon_synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:194:in `invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:183:in `invoke'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:160:in `invoke_task'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:116:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:116:in `block in top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:125:in `run_with_threads'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:110:in `top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:83:in `block in run'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:186:in `standard_exception_handling'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:80:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/exe/rake:27:in `<top (required)>'
/opt/gitlab/embedded/bin/rake:23:in `load'
/opt/gitlab/embedded/bin/rake:23:in `<top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in `load'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in `kernel_load'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:28:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:463:in `exec'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor.rb:387:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:27:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/base.rb:466:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:18:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/exe/bundle:30:in `block in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/friendly_errors.rb:124:in `with_friendly_errors'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/exe/bundle:22:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:elastic:create_empty_index
What is the expected correct behavior?
gitlab-rake gitlab:elastic:create_empty_index
Index created
Relevant logs
Relevant logs
Details of package version
gitlab-rake gitlab:env:info
System information
System:
Proxy: no
Current User: git
Using RVM: no
Ruby Version: 2.6.5p114
Gem Version: 2.7.10
Bundler Version:1.17.3
Rake Version: 12.3.3
Redis Version: 5.0.7
Git Version: 2.24.1
Sidekiq Version:5.2.7
Go Version: unknown
GitLab information
Version: 12.7.2-ee
Revision: be1bc017799
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: PostgreSQL
DB Version: 10.9
URL: <redacted>
HTTP Clone URL: <redacted>
SSH Clone URL: <redacted>
Elasticsearch: yes
Geo: no
Using LDAP: no
Using Omniauth: yes
Omniauth Providers: saml
GitLab Shell
Version: 11.0.0
Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories
GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell
Git: /opt/gitlab/embedded/bin/git
Environment details
- Operating System:
Amazon Linux AMI release 2018.03
- Installation Target:
- VM:
AWS
- VM:
- Installation Type, remove incorrect values:
- Upgrade from version
12.5.5
- Upgrade from version
- Is there any other software running on the machine:
No
- Is this a single or multiple node installation?
No
- Resources
- CPU:
4 VCPU
- Memory total:
16 GiB
- CPU: