S3 Cache with AssumeRoleWithWebIdentity fails with incorrect STS URL in AWS Beijing region
Summary
We are running GitLab runners in EKS in the EU, US, and China AWS regions with S3 caching. All works correctly in EU and US regions, but China gives us the following error:
ERROR: error while generating S3 pre-signed URL error=Post https://sts.cn-north-1.amazonaws.com?Action=AssumeRoleWithWebIdentity&RoleArn=arn%3Aaws-cn%3Aiam%3A%3A<AWS ACCT NUMBER REDACTED>%3Arole%2Feks-ops-cn-gl-runners-secrets-csi-driver&RoleSessionName=1625589185585289293&Version=2011-06-15&WebIdentityToken= <REDACTED>: dial tcp: lookup sts.cn-north-1.amazonaws.com on 172.20.0.10:53: no such host
As can be seen, the URL generated is: https://sts.cn-north-1.amazonaws.com
, but that should actually be: https://sts.cn-north-1.amazonaws.com.cn
We have worked with AWS support and after running some tests, they believe this to be an issue with the GitLab runner pod.
Steps to reproduce
Set up S3 cache details in kubernetes chart (truncated to show only relevant section)
apiVersion: apps/v1
kind: Deployment
spec:
template:
spec:
containers:
- command:
- /bin/bash
- /scripts/entypoint
env:
- name: CACHE_TYPE
value: s3
- name: CACHE_PATH
value: gitlab-caches
- name: CACHE_SHARED
value: "true"
- name: CACHE_S3_SERVER_ADDRESS
value: s3.amazonaws.com.cn
- name: CACHE_S3_BUCKET_NAME
value: gitlab-cache
- name: CACHE_S3_BUCKET_LOCATION
value: cn-north-1
Gitlab Runner service account is annotated with:
apiVersion: v1
kind: ServiceAccount
metadata:
annotations:
eks.amazonaws.com/role-arn: arn:aws-cn:iam::111111111111:role/pod-role
.gitlab-ci.yml
#####
# Global
#####
cache: &global_cache
key: "${CI_COMMIT_REF_SLUG}-${CI_COMMIT_SHA}"
paths:
- ${TF_DIR}/.terraform/modules
- ${TF_DIR}/.terraform/plugins
Actual behavior
Cache fails as the STS URL that's generated is incorrect for China
Expected behavior
Cacheing should succeed with a successful connection to the S3 bucket
Relevant logs and/or screenshots
job log
Add the job log
Environment description
We are running custom shared group runners on GitLab.com which are running on AWS EKS 1.20
Used GitLab Runner version
Running with gitlab-runner 14.0.1 (c1edb478)
on gitlab-runner-gitlab-runner-6db4c58df5-999h9 39LNSLEh