Elasticsearch with AWS still getting throttled due to instance profile credentials
In https://gitlab.com/gitlab-org/gitlab-ee/issues/3650 we noted a problem with the Rails and Sidekiq processes having too many elasticsearch clients, leading to many instance profile endpoint requests and throttling/rate-limited. !3432 (merged) fixed that issue, but it turns out we have another problem:
When we shell out to bin/elastic_repo_indexer or https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer , the subprocess does an instance profile endpoint call. Here's a discussion we had about it in the MR that implemented instance profile support: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/2020/diffs#note_31397249
(/cc @mgresko)
I think we need another MR that retrieves the cached credentials in gitlab-rails and passes them to the ruby/go subprocess explicitly, rather than having those subprocesses do instance profile endpoint calls. /cc @victorwu for scheduling, @smcgivern for sanity checking.
We might also want to add a control to completely disable instance profile support.