S3 Cache not working with IRSA in EKS 1.18
Summary
S3 Cache not working for Gitlab Runner in EKS 1.18 with IRSA. This seems to happen since upgrading eks from 1.17 to 1.18
Steps to reproduce
.gitlab-ci.yml
image: registry.gitlab.com/gitlab-org/terraform-images/stable:latest
variables:
TF_ROOT: ${CI_PROJECT_DIR}/${SERVICE}
cache:
key: c11h-terraform-${TEAM}-${SERVICE}
paths:
- ${TF_ROOT}/.terraformActual behavior
Expected behavior
Relevant logs and/or screenshots
gitlab-1-ops-gitlab-runner-8494b66d66-7pznk gitlab-1-ops-gitlab-runner ERROR: error while generating S3 pre-signed URL error=Post sts.eu-central-1.amazonaws.com?Action=AssumeRoleWithWebIdentity&RoleArn=arn%3Aaws%3Aiam%3A%3A026861505133%3Arole%2Firsa-gitlab-1-ops-manager&RoleSessionName=1603453288209630053&Version=2011-06-15&WebIdentityToken=eyJhbGciOiJSUzI1NiIsImtpZCI6IjhlMWJiNzBkY2NlMGNlZjJjYmUzYmJlYTMyNjZkMDQ3MWUzOTc3ZTgifQ.eyJhdWQiOlsic3RzLmFtYXpvbmF3cy5jb20iXSwiZXhwIjoxNjAzNTM5NTU4LCJpYXQiOjE2MDM0NTMxNTgsImlzcyI6Imh0dHBzOi8vb2lkYy5la3MuZXUtY2VudHJhbC0xLmFtYXpvbmF3cy5jb20vaWQvQzg5OTJCNjNGRUEyMTVFQzc0REFCMDZEMTFDNTQzMUMiLCJrdWJlcm5ldGVzLmlvIjp7Im5hbWVzcGFjZSI6Im9wcy1rOHNydW5uZXIiLCJwb2QiOnsibmFtZSI6ImdpdGxhYi0xLW9wcy1naXRsYWItcnVubmVyLTg0OTRiNjZkNjYtN3B6bmsiLCJ1aWQiOiI3NGY3MzIwMi1iMTc3LTRhNjItODdmNS1jYTMyZDU1N2Y5NGYifSwic2VydmljZWFjY291bnQiOnsibmFtZSI6ImdpdGxhYi0xLW9wcy1tYW5hZ2VyIiwidWlkIjoiNGFjZWE1YWEtMzA2My00MTM3LWE1NzMtZGVlMzcyNGE0NDcyIn19LCJuYmYiOjE2MDM0NTMxNTgsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpvcHMtazhzcnVubmVyOmdpdGxhYi0xLW9wcy1tYW5hZ2VyIn0.VuWQipMrgopZVZs15ZZY8oKGt4h0bYmW3kb5Fja9PP3Gi9jYrrEGVenNUfgYwnHo9rhxZ4ARrOFq64PFmaPEgNLtvyabCKywGOpuebOi416knHuM0FagUIwulollZYCPXw7b5BXL5Pv3AH493cIApqAq0-PI1LpynjioIkP29zXdNIost4XQHTC5q_oQXr7jRAMn54Zj1DxTlwqyT5AjZS7_H8kCQHzPh2RmkjQu3-VLlyIGodzYqQrtKPJWQIywLgv4znTOcX1zG-XyijcQX2cptUPWNSzmuBQONP2YpcsnA-u0pyd5Nbk8dgJnzKiUhiEh58nkhFyi8CbY3bTwdA: unsupported protocol scheme ""
job log
Add the job logEnvironment description
GitLab Enterprise Edition 13.5.1-ee
git@gitlab-server-webservice-5bdc88b9b-sbc9b:/$ cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
NAME="Debian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
VERSION_CODENAME=stretch
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"config.toml contents
concurrent = 50
check_interval = 5
log_level = "warn"
[session_server]
session_timeout = 1800
[[runners]]
name = "gitlab-1-ops-gitlab-runner-8494b66d66-7pznk"
output_limit = 8192
request_concurrency = 1
url = "https://ourgit.com/"
token = "token"
executor = "kubernetes"
[runners.custom_build_dir]
[runners.cache]
Type = "s3"
Path = "ops"
Shared = true
[runners.cache.s3]
ServerAddress = "s3.amazonaws.com"
BucketName = "secretbucket"
BucketLocation = "eu-central-1"
[runners.cache.gcs]
[runners.cache.azure]
[runners.kubernetes]
host = ""
bearer_token_overwrite_allowed = false
image = "docker:stable"
namespace = "ops-k8srunner"
namespace_overwrite_allowed = ""
privileged = true
cpu_limit = "6"
cpu_limit_overwrite_max_allowed = "16"
cpu_request = "128m"
cpu_request_overwrite_max_allowed = "16"
memory_limit = "28Gi"
memory_limit_overwrite_max_allowed = "48Gi"
memory_request = "512Mi"
memory_request_overwrite_max_allowed = "48Gi"
service_cpu_limit = "4"
service_cpu_request = "128m"
service_memory_limit = "10000Mi"
service_memory_request = "512Mi"
helper_cpu_limit = "4"
helper_cpu_request = "128m"
helper_memory_limit = "10000Mi"
helper_memory_request = "512Mi"
poll_timeout = 1200
service_account = "gitlab-1-ops-executor"
service_account_overwrite_allowed = ""
pod_annotations_overwrite_allowed = ""
[runners.kubernetes.node_selector]
autoscaling-node = "true"
[runners.kubernetes.node_tolerations]
"autoscaling-node=false" = "NoExecute"
[runners.kubernetes.affinity]
[runners.kubernetes.pod_security_context]
[runners.kubernetes.volumes] Used GitLab Runner version
Version: 13.5.0
Git revision: ece86343
Git branch: 13-5-stable
GO version: go1.13.8
Built: 2020-10-20T12:05:22+0000
OS/Arch: linux/amd64
Possible fixes
https://github.com/gitlabhq/gitlab-runner/blob/master/cache/s3/minio.go