Always pass the AWS credentails from sidekiq to indexer for AWS services autentication
Problem to solve
This issue stems from a community contribution !24926 (closed), which is to provide the full range of authentication options provided by AWS. The solution would allow applications running in AWS ECS environment to obtain the credentials that are needed for service authentication. For more information about AWS ECS credential management strategy, please refer to this document, !24926 (closed).
In the discussion in !24926 (closed), @spyoungtech and @kusuriya have tested passing required environment variables to indexer, which in turn can fetch the credentials via endpoint in the ECS agent container. This solution works in our users setup. When reviewing this solution, @nick.thomas indicated there could one rate limiting related issue and suggested passing the credentials from sidekiq side,
An issue we've seen in the past is that having lots of parallel
gitlab-elasticsearch-indexerprocesses try to pull their own credentials or read from zeroconfig endpoints leads to hitting rate limits. So it's much better to do this kind of work in sidekiq and pass the results intogitlab-elasticsearch-indexervia the environment.
@DylanGriffith also pointed out another benefit of doing this,
Additional benefit to moving to Ruby would be single source of truth since we need to do this in Ruby anyway.
Intended users
Further details
Proposal
We could try to fetch the credentials from sidekiq in case there are no statically configured credentials, then pass them when invoking gitlab-elasticsearch-indexer.
Permissions and Security
Documentation
We need to document what AWS environment we support.
Availability & Testing
It would be good if we can test it in the desired AWS environment, like EC2 and ECS.