Read AWS credentials from environment and IAM roles across all components
Description
Most AWS service libraries are capable of reading AWS IAM credentials from environment variables and/or the EC2 metadata service running in EC2. This allows easy, hands off configuration of credentials especially when running on EC2 instances where you merely attach an IAM role to the instance, and code inside the EC2 instance can automatically start using the new credentials. In many cases (boto, Java AWS-SDK, Go AWS-SDK) using the default credential provider chain will search a series of well-known credential providers and "just work" if credentials are provided via one of these mechanisms.
Fetching credentials from the EC2 metadata service also offers a security benefit. AWS rotates these credentials automatically every six to twelve hours providing a big improvement over static credentials stored in a config file that must be manually rotated.
It was a surprising and un-intuitive experience to spin up Gitlab 10.4 and see gitlab-runner could use IAM roles to create docker-machine based runners, but the distributed caching service failed to function (fixed in 10.6, specifically gitlab-runner!646 (merged).) Similarly the Gitlab Registry was capable of using an IAM role but configuring S3-backed Artifact object storage and running gitlab-rake gitlab:artifacts:migrate
returned a bunch of errors:
ERROR -- : Failed to transfer artifacts of 47 with error: Missing required arguments: aws_access_key_id, aws_secret_access_key
ERROR -- : Failed to transfer artifacts of 61 with error: Missing required arguments: aws_access_key_id, aws_secret_access_key
ERROR -- : Failed to transfer artifacts of 65 with error: Missing required arguments: aws_access_key_id, aws_secret_access_key
Proposal
Support reading from the EC2 metadata service across all components of Gitlab for a more consistent experience. At least gitlab-rake is currently lacking support as evidenced by the above Artifacts object storage failure.
Links / references
(Boto credentials documentation)[https://boto3.readthedocs.io/en/latest/guide/configuration.html#configuring-credentials]
(AWS SDK for Go credentials documentation)[https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html#specifying-credentials]
(AWS SDK for Java)[https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html#credentials-default]
(Fog-aws credential fetcher)[https://github.com/fog/fog-aws/blob/master/lib/fog/aws/credential_fetcher.rb]