Skip to content
GitLab
Next
    • GitLab: the DevOps platform
    • Explore GitLab
    • Install GitLab
    • How GitLab compares
    • Get started
    • GitLab docs
    • GitLab Learn
  • Pricing
  • Talk to an expert
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
    Projects Groups Snippets
  • Sign up now
  • Login
  • Sign in / Register
  • gitlab-runner gitlab-runner
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
    • Locked Files
  • Issues 2,609
    • Issues 2,609
    • List
    • Boards
    • Service Desk
    • Milestones
    • Iterations
    • Requirements
  • Merge requests 107
    • Merge requests 107
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
    • Test Cases
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Packages and registries
    • Packages and registries
    • Container Registry
  • Monitor
    • Monitor
    • Metrics
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Code review
    • Insights
    • Issue
    • Repository
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • GitLab.orgGitLab.org
  • gitlab-runnergitlab-runner
  • Issues
  • #27152
Closed
Open
Issue created Oct 23, 2020 by Michael vanAppeldorn@michael.vanappeldorn

S3 Cache not working with IRSA in EKS 1.18

Summary

S3 Cache not working for Gitlab Runner in EKS 1.18 with IRSA. This seems to happen since upgrading eks from 1.17 to 1.18

Steps to reproduce

.gitlab-ci.yml
image: registry.gitlab.com/gitlab-org/terraform-images/stable:latest
variables:
  TF_ROOT: ${CI_PROJECT_DIR}/${SERVICE}

cache:
  key: c11h-terraform-${TEAM}-${SERVICE}
  paths:
    - ${TF_ROOT}/.terraform

Actual behavior

Expected behavior

Relevant logs and/or screenshots

gitlab-1-ops-gitlab-runner-8494b66d66-7pznk gitlab-1-ops-gitlab-runner ERROR: error while generating S3 pre-signed URL error=Post sts.eu-central-1.amazonaws.com?Action=AssumeRoleWithWebIdentity&RoleArn=arn%3Aaws%3Aiam%3A%3A026861505133%3Arole%2Firsa-gitlab-1-ops-manager&RoleSessionName=1603453288209630053&Version=2011-06-15&WebIdentityToken=eyJhbGciOiJSUzI1NiIsImtpZCI6IjhlMWJiNzBkY2NlMGNlZjJjYmUzYmJlYTMyNjZkMDQ3MWUzOTc3ZTgifQ.eyJhdWQiOlsic3RzLmFtYXpvbmF3cy5jb20iXSwiZXhwIjoxNjAzNTM5NTU4LCJpYXQiOjE2MDM0NTMxNTgsImlzcyI6Imh0dHBzOi8vb2lkYy5la3MuZXUtY2VudHJhbC0xLmFtYXpvbmF3cy5jb20vaWQvQzg5OTJCNjNGRUEyMTVFQzc0REFCMDZEMTFDNTQzMUMiLCJrdWJlcm5ldGVzLmlvIjp7Im5hbWVzcGFjZSI6Im9wcy1rOHNydW5uZXIiLCJwb2QiOnsibmFtZSI6ImdpdGxhYi0xLW9wcy1naXRsYWItcnVubmVyLTg0OTRiNjZkNjYtN3B6bmsiLCJ1aWQiOiI3NGY3MzIwMi1iMTc3LTRhNjItODdmNS1jYTMyZDU1N2Y5NGYifSwic2VydmljZWFjY291bnQiOnsibmFtZSI6ImdpdGxhYi0xLW9wcy1tYW5hZ2VyIiwidWlkIjoiNGFjZWE1YWEtMzA2My00MTM3LWE1NzMtZGVlMzcyNGE0NDcyIn19LCJuYmYiOjE2MDM0NTMxNTgsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpvcHMtazhzcnVubmVyOmdpdGxhYi0xLW9wcy1tYW5hZ2VyIn0.VuWQipMrgopZVZs15ZZY8oKGt4h0bYmW3kb5Fja9PP3Gi9jYrrEGVenNUfgYwnHo9rhxZ4ARrOFq64PFmaPEgNLtvyabCKywGOpuebOi416knHuM0FagUIwulollZYCPXw7b5BXL5Pv3AH493cIApqAq0-PI1LpynjioIkP29zXdNIost4XQHTC5q_oQXr7jRAMn54Zj1DxTlwqyT5AjZS7_H8kCQHzPh2RmkjQu3-VLlyIGodzYqQrtKPJWQIywLgv4znTOcX1zG-XyijcQX2cptUPWNSzmuBQONP2YpcsnA-u0pyd5Nbk8dgJnzKiUhiEh58nkhFyi8CbY3bTwdA: unsupported protocol scheme ""

job log
Add the job log

Environment description

GitLab Enterprise Edition 13.5.1-ee

git@gitlab-server-webservice-5bdc88b9b-sbc9b:/$ cat /etc/os-release 
PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
NAME="Debian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
VERSION_CODENAME=stretch
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
config.toml contents
concurrent = 50                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     
check_interval = 5                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
log_level = "warn"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    
[session_server]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    
  session_timeout = 1800                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    
[[runners]]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         
  name = "gitlab-1-ops-gitlab-runner-8494b66d66-7pznk"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              
  output_limit = 8192                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               
  request_concurrency = 1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
  url = "https://ourgit.com/"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     
  token = "token"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    
  executor = "kubernetes"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
  [runners.custom_build_dir]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        
  [runners.cache]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
    Type = "s3"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     
    Path = "ops"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    
    Shared = true                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
    [runners.cache.s3]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              
      ServerAddress = "s3.amazonaws.com"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
      BucketName = "secretbucket"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          
      BucketLocation = "eu-central-1"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               
    [runners.cache.gcs]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             
    [runners.cache.azure]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
  [runners.kubernetes]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              
    host = ""                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       
    bearer_token_overwrite_allowed = false                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          
    image = "docker:stable"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         
    namespace = "ops-k8srunner"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     
    namespace_overwrite_allowed = ""                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                
    privileged = true                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               
    cpu_limit = "6"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 
    cpu_limit_overwrite_max_allowed = "16"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          
    cpu_request = "128m"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
    cpu_request_overwrite_max_allowed = "16"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        
    memory_limit = "28Gi"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
    memory_limit_overwrite_max_allowed = "48Gi"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     
    memory_request = "512Mi"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        
    memory_request_overwrite_max_allowed = "48Gi"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
    service_cpu_limit = "4"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         
    service_cpu_request = "128m"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    
    service_memory_limit = "10000Mi"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                
    service_memory_request = "512Mi"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                
    helper_cpu_limit = "4"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          
    helper_cpu_request = "128m"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     
    helper_memory_limit = "10000Mi"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 
    helper_memory_request = "512Mi"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 
    poll_timeout = 1200                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             
    service_account = "gitlab-1-ops-executor"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       
    service_account_overwrite_allowed = ""                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          
    pod_annotations_overwrite_allowed = ""                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          
    [runners.kubernetes.node_selector]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              
      autoscaling-node = "true"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     
    [runners.kubernetes.node_tolerations]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
      "autoscaling-node=false" = "NoExecute"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        
    [runners.kubernetes.affinity]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
    [runners.kubernetes.pod_security_context]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       
    [runners.kubernetes.volumes]            

Used GitLab Runner version

Version:      13.5.0
Git revision: ece86343
Git branch:   13-5-stable
GO version:   go1.13.8
Built:        2020-10-20T12:05:22+0000
OS/Arch:      linux/amd64

Possible fixes

https://github.com/gitlabhq/gitlab-runner/blob/master/cache/s3/minio.go

Edited Oct 23, 2020 by Michael vanAppeldorn
Assignee
Assign to
Time tracking