ERROR: error while generating S3 pre-signed URL 400 Bad Request

Summary

Error generating an S3 pre-signed URL when using IAM role. Returns a 400 Bad Request error.

Steps to reproduce

I created an EKS 1.20 cluster. Create an IAM role with admin access policy for testing. Annotated the role in helm values.yaml file. Also tried creating my own service account, service account role and attaching the IAM policy.

.gitlab-ci.yml 
.node:
  image: mhart/alpine-node:12
  before_script:
    - export AWS_REGION=$AWS_DEFAULT_REGION
    - apk add jq
    - apk add py3-pip
    - apk add git
    - apk add bash
    - apk add openssh
    - apk add --update make
    - pip3 install --upgrade pip
    - pip3 install awscli
    - eval $(aws sts assume-role --role-arn "arn:aws:iam::$AWS_ACCOUNT_ID:role/DeploymentRole" --role-session-name terraform_deploy | jq -r '.Credentials | @sh "export AWS_SESSION_TOKEN=\(.SessionToken)\nexport AWS_ACCESS_KEY_ID=\(.AccessKeyId)\nexport AWS_SECRET_ACCESS_KEY=\(.SecretAccessKey) "')

install_dependencies:
  stage: install_dependencies
  extends: .node
  cache: 
    key: $CI_COMMIT_REF_SLUG-$CI_PROJECT_DIR
    paths:
      - node_modules/
  script:
    - npm ci
  rules:
    - if: '$CI_MERGE_REQUEST_ID == null'
      changes:
        - package-lock.json

Actual behavior

Job succeeds with following message: No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted.

Error in the pod logs is: [31;1mERROR: error while generating S3 pre-signed URL [0;m [31;1merror[0;m=400 Bad Request

Expected behavior

Expect to see a URL and cache uploaded to S3 bucket.

Relevant logs and/or screenshots

job log 
Registration attempt 1 of 30
Runtime platform                                  [0;m  arch[0;m=amd64 os[0;m=linux pid[0;m=12 revision[0;m=7f7a4bb0 version[0;m=13.11.0
[0;33mWARNING: Running in user-mode.                    [0;m 
[0;33mWARNING: The user-mode requires you to manually start builds processing:[0;m 
[0;33mWARNING: $ gitlab-runner run                      [0;m 
[0;33mWARNING: Use sudo for system-mode:                [0;m 
[0;33mWARNING: $ sudo gitlab-runner...                  [0;m 
                                                  [0;m 
Registering runner... succeeded                   [0;m  runner[0;m=jMiBSUCx
Merging configuration from template file "/configmaps/config.template.toml"[0;m 
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded![0;m 
Runtime platform                                  [0;m  arch[0;m=amd64 os[0;m=linux pid[0;m=1 revision[0;m=7f7a4bb0 version[0;m=13.11.0
Starting multi-runner from /home/gitlab-runner/.gitlab-runner/config.toml...[0;m  builds[0;m=0
[0;33mWARNING: Running in user-mode.                    [0;m 
[0;33mWARNING: Use sudo for system-mode:                [0;m 
[0;33mWARNING: $ sudo gitlab-runner...                  [0;m 
                                                  [0;m 
Configuration loaded                              [0;m  builds[0;m=0
Metrics server listening                          [0;m  address[0;m=:9252 builds[0;m=0
[session_server].listen_address not defined, session endpoints disabled[0;m  builds[0;m=0
Checking for jobs... received                     [0;m  job[0;m=1460841205 repo_url[0;m=https://gitlab.com/repo.git runner[0;m=HXezoxyD
[0;33mWARNING: Pulling GitLab Runner helper image from Docker Hub. Helper image is migrating to registry.gitlab.com, for more information see https://docs.gitlab.com/runner/configuration/advanced-configuration.html#migrate-helper-image-to-registrygitlabcom[0;m  [0;33mjob[0;m=1460841205 [0;33mproject[0;m=23197539 [0;33mrunner[0;m=HXezoxyD
[31;1mERROR: error while generating S3 pre-signed URL   [0;m  [31;1merror[0;m=400 Bad Request

Environment description

Service Account:

apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::11111111:role/test-runner-role
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"ServiceAccount","metadata":{"annotations":{},"name":"gitlab-runner-service-account","namespace":"gitlab-managed-apps"}}
  creationTimestamp: "2021-07-29T02:16:04Z"
  name: gitlab-runner-service-account
  namespace: gitlab-managed-apps
  resourceVersion: "2429501"
  uid: cd76b72f-d86e-456c-a32b
secrets:
- name: gitlab-runner-service-account-token-asdf
config.toml contents 
rbac:
  create: false
  serviceAccountName: gitlab-runner-service-account
  clusterWideAccess: false
  serviceAccountAnnotations :
    eks.amazonaws.com/role-arn: "arn:aws:iam::11111111111:role/test-runner-role"
runners:  
  runUntagged: true
  config: |
    [[runners]]
      [runners.kubernetes]
        namespace = "{{.Release.Namespace}}"
        image = "alpine/git"
        memory_limit = "7Gi"
      [runners.cache]
        Type = "s3"
        Path = "path"
        Shared = true
        [runners.cache.s3]
          ServerAddress = "s3.amazonaws.com"
          BucketName = "gitlab-runner-cache"
          BucketLocation = "us-east-2"

  serviceAccountName: "gitlab-runner-service-account"

Used GitLab Runner version

Possible fixes

Edited by Tim Fendt