Fix Role ARN support with S3 Express buckets

What does this MR do?

When the runner requests temporary scoped S3 credentials, S3 Express buckets need the s3express:CreateSession policy in order to access them. The runner reports this error if that permission is not available:

operation error S3: HeadObject, get identity: get credentials: operation error S3: CreateSession, https response error StatusCode: 403

Add this permission if we detect an S3 Express bucket with the --x-s3 suffix (https://docs.aws.amazon.com/AmazonS3/latest/userguide/directory-buckets-overview.html). This is what the AWS SDK library does too.

Why was this MR needed?

What's the best way to test this MR?

  1. Set up an S3 Express bucket: https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-express-getting-started.html
  2. In config.toml, you should have something like:
  [runners.cache]
    Type = "s3"
    MaxUploadedArchiveSize = 0
    [runners.cache.s3]
    RoleARN = "arn:aws:iam::<ACCOUNT ID>:role/your-example-role"
    BucketName = "your-s3-express-bucket--usw2-az1--x-s3"
    BucketLocation = "us-west-2"
    DualStack = false
  1. Run a CI job that uses the cache:
default:
  script:
    - echo "hello world" > test.txt
  cache:
    paths:
      - test.txt
  artifacts:
    paths:
      - test.txt

What are the relevant issue numbers?

#38484 (closed)

Edited by Stan Hu

Merge request reports

Loading