Helper side-car container fails to upload cache into s3 bucket from another account

Summary

Flow: we have a k8s runner in AWS account A which assumes role from AWS account B, and we would like the runner's cache to be uploaded to S3 bucket in account A. It fails.

What we have:

  • self-hosted GitLab instance (17.1) installed on EC2 machine.
  • Kubernetes runner deployed in EKS cluster A under AWS account account_A
  • AWS account account_B with IAM role runner-role
  • runner deployed in AWS account A assumes role in account B. Works fine. Runner has sufficient rights to operate in in target account B.
  • when Kubernetes runner finishes script execution, helper image creates cache, archives it and tries to upload it to S3 bucket.

Problem

Error it receives trying to upload cache: "FATAL: received: 400 Bad Request"

At the same time, runner has sufficient rights to operate with cache bucket. The following snippet added to script definition works:

touch empty-file
aws s3 cp empty-file s3://cache-bucket-used-by-runner

Additional Info

AWS resources involved into the flow:

  • S3 bucket
  • S3 bucket policy
  • Assume Role policy for IAM role which runner uses
  • IAM policy attached to IAM role used by runner
  • OIDC provider in target environment account_B which links to OIDC provider in account_A
  • KMS key
  • KMS key policy

All that seems to be OK, since the runner has rights to upload files as part of its main job.

In scenario where account_A is the same as account_B, means when the role assumed by runner stays in the same account where runner executor is deployed, cache upload works fine as well.

Also tried to assign unlimited IAM rights everywhere for the mentioned resources (s3 bucket policy, KMS key policy, runner policy). Same outcome, so I'm quite confident that the issue is not on IAM side.

Setup Details

Runner is installed using official helm chart.

Default config.

Extra vars are:

extraEnv:
  CACHE_TYPE: s3
  CACHE_SHARED: true
  CACHE_S3_SERVER_ADDRESS: s3.amazonaws.com
  CACHE_S3_AUTHENTICATION_TYPE: IAM
  CACHE_S3_SERVER_SIDE_ENCRYPTION: KMS
  CACHE_S3_BUCKET_NAME: <SKIPPED>
  CACHE_S3_BUCKET_LOCATION: eu-central-1
  CACHE_S3_SERVER_SIDE_ENCRYPTION_KEY_ID: <SKIPPED>
  KUBERNETES_SERVICE_ACCOUNT: <SKIPPED>

Tried the same setup with config.toml file. Same outcome.

Edited by Konstantin Dobroliubov