AWS IAM Authentication Method not working since Gitlab runner v16

Status Update - 2024-07-18

All the tests ran by @ratchade to reproduce this issue were not conclusive. Both instance profile and IRSA for AWS were tested.

Do not hesitate to re open the issue and provide new information if you are still experiencing this issue 🙇🏿

Summary

It is already mentioned in the open issue #16097 (comment 1469400739) but since the issue itself is another one there I would like to open a dedicated issue.

Since we updated our Gitlab runner to v16 S3 caching cannot be used. We see this error in the job logs (Gitlab runner 16.3.0 - helm chart 0.56):

$ touch "$CI_PROJECT_DIR/testcache"
Saving cache for successful job
00:01
Creating cache test-cache-non_protected...
.../testcache: found 1 matching artifact files and directories 
No URL provided, cache will not be uploaded to shared cache server. Cache will be stored only locally. 
Created cache
Cleaning up project directory and file based variables
00:00
Job succeeded

Just downgraded to version 15.11.1 (chart v0.52.1) without changing the config:

$ touch "$CI_PROJECT_DIR/testcache"
Saving cache for successful job
00:00
Creating cache test-cache-non_protected...
.../testcache: found 1 matching artifact files and directories 
Uploading cache.zip to https://
Created cache
Cleaning up project directory and file based variables
00:01
Job succeeded

Steps to reproduce

use Gitlab runner IAM configuration:

      [runners.cache]
        Type = "s3"
        Shared = true
        [runners.cache.s3]
          ServerAddress = "s3.amazonaws.com"
          BucketName = "gitlab-cache-bucket"
          BucketLocation = "eu-west-1"
          AuthenticationType = "iam"
          Insecure = false

stages:
  - test

test-cache:
  stage: test
  image: alpine
  script:
    - touch "$CI_PROJECT_DIR/testcache"
  cache:
    - key: test-cache
      policy: pull-push
      paths:
        - "$CI_PROJECT_DIR/testcache"

Environment description

We do use Gitlab.com but use our custom runners on Kubernetes installed via Helm chart.