AWS credential exposure in failed GitLab Runner cache upload to S3
From: https://gitlab.zendesk.com/agent/tickets/128877
Hello,
We've noticed that if the cache cache upload to S3 fails, the retries expose X-Amz-Credential and X-Amz-Signature query string parameters. Please note that the line prefixed with FATAL is scrubbed properly.
Anonymized job trace:
Uploading cache.zip to https://MYBUCKET.s3-eu-west-1.amazonaws.com/worker/cache/project/PROJECTID/default
WARNING: Retrying... error=Put https://MYBUCKET.s3-eu-west-1.amazonaws.com/worker/cache/project/PROJECTID/default?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AWSCREDENTIALS%2F20190807%2Feu-west-1%2Fs3%2Faws4_request&X-Amz-Date=20190807T130444Z&X-Amz-Expires=3600&X-Amz-Security-Token=AWSSECURITYTOKEN&X-Amz-SignedHeaders=host&X-Amz-Signature=AWSSIGNATURE: write tcp 172.17.0.2:60024->52.218.53.219:443: write: connection reset by peer
Uploading cache.zip to https://MYBUCKET.s3-eu-west-1.amazonaws.com/worker/cache/project/PROJECTID/default
WARNING: Retrying... error=Put https://MYBUCKET.s3-eu-west-1.amazonaws.com/worker/cache/project/PROJECTID/default?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AWSCREDENTIALS%2F20190807%2Feu-west-1%2Fs3%2Faws4_request&X-Amz-Date=20190807T130444Z&X-Amz-Expires=3600&X-Amz-Security-Token=AWSSECURITYTOKEN&X-Amz-SignedHeaders=host&X-Amz-Signature=AWSSIGNATURE: write tcp 172.17.0.2:33830->52.218.109.32:443: write: connection reset by peer
Uploading cache.zip to https://MYBUCKET.s3-eu-west-1.amazonaws.com/worker/cache/project/PROJECTID/default
FATAL: Put https://MYBUCKET.s3-eu-west-1.amazonaws.com/worker/cache/project/PROJECTID/default?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=[FILTERED]&X-Amz-Date=20190807T130444Z&X-Amz-Expires=3600&X-Amz-Security-Token=AWSSECURITYTOKEN&X-Amz-SignedHeaders=host&X-Amz-Signature=[FILTERED] write tcp 172.17.0.2:33832->52.218.109.32:443: write: connection reset by peer
Failed to create cache
I believe I've narrowed the issue down to the cache archiver helper https://gitlab.com/gitlab-org/gitlab-runner/blob/v12.0.2/commands/helpers/cache_archiver.go To me it looks like the error message returned by the upload() function is not being scrubbed. A little background on our setup, we're currently running GitLab Runner 12.0.2 with an S3 cache backend and the Docker executor.
Regards, Jan P
Edited by Darren Eastman