ERROR: error while generating S3 pre-signed URL 400 Bad Request
Summary
Error generating an S3 pre-signed URL when using IAM role. Returns a 400 Bad Request error.
Steps to reproduce
I created an EKS 1.20 cluster. Create an IAM role with admin access policy for testing. Annotated the role in helm values.yaml file. Also tried creating my own service account, service account role and attaching the IAM policy.
.gitlab-ci.yml
.node:
image: mhart/alpine-node:12
before_script:
- export AWS_REGION=$AWS_DEFAULT_REGION
- apk add jq
- apk add py3-pip
- apk add git
- apk add bash
- apk add openssh
- apk add --update make
- pip3 install --upgrade pip
- pip3 install awscli
- eval $(aws sts assume-role --role-arn "arn:aws:iam::$AWS_ACCOUNT_ID:role/DeploymentRole" --role-session-name terraform_deploy | jq -r '.Credentials | @sh "export AWS_SESSION_TOKEN=\(.SessionToken)\nexport AWS_ACCESS_KEY_ID=\(.AccessKeyId)\nexport AWS_SECRET_ACCESS_KEY=\(.SecretAccessKey) "')
install_dependencies:
stage: install_dependencies
extends: .node
cache:
key: $CI_COMMIT_REF_SLUG-$CI_PROJECT_DIR
paths:
- node_modules/
script:
- npm ci
rules:
- if: '$CI_MERGE_REQUEST_ID == null'
changes:
- package-lock.json
Actual behavior
Job succeeds with following message: No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted.
Error in the pod logs is: [31;1mERROR: error while generating S3 pre-signed URL [0;m [31;1merror[0;m=400 Bad Request
Expected behavior
Expect to see a URL and cache uploaded to S3 bucket.
Relevant logs and/or screenshots
job log
Registration attempt 1 of 30
Runtime platform [0;m arch[0;m=amd64 os[0;m=linux pid[0;m=12 revision[0;m=7f7a4bb0 version[0;m=13.11.0
[0;33mWARNING: Running in user-mode. [0;m
[0;33mWARNING: The user-mode requires you to manually start builds processing:[0;m
[0;33mWARNING: $ gitlab-runner run [0;m
[0;33mWARNING: Use sudo for system-mode: [0;m
[0;33mWARNING: $ sudo gitlab-runner... [0;m
[0;m
Registering runner... succeeded [0;m runner[0;m=jMiBSUCx
Merging configuration from template file "/configmaps/config.template.toml"[0;m
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded![0;m
Runtime platform [0;m arch[0;m=amd64 os[0;m=linux pid[0;m=1 revision[0;m=7f7a4bb0 version[0;m=13.11.0
Starting multi-runner from /home/gitlab-runner/.gitlab-runner/config.toml...[0;m builds[0;m=0
[0;33mWARNING: Running in user-mode. [0;m
[0;33mWARNING: Use sudo for system-mode: [0;m
[0;33mWARNING: $ sudo gitlab-runner... [0;m
[0;m
Configuration loaded [0;m builds[0;m=0
Metrics server listening [0;m address[0;m=:9252 builds[0;m=0
[session_server].listen_address not defined, session endpoints disabled[0;m builds[0;m=0
Checking for jobs... received [0;m job[0;m=1460841205 repo_url[0;m=https://gitlab.com/repo.git runner[0;m=HXezoxyD
[0;33mWARNING: Pulling GitLab Runner helper image from Docker Hub. Helper image is migrating to registry.gitlab.com, for more information see https://docs.gitlab.com/runner/configuration/advanced-configuration.html#migrate-helper-image-to-registrygitlabcom[0;m [0;33mjob[0;m=1460841205 [0;33mproject[0;m=23197539 [0;33mrunner[0;m=HXezoxyD
[31;1mERROR: error while generating S3 pre-signed URL [0;m [31;1merror[0;m=400 Bad Request
Environment description
Service Account:
apiVersion: v1
kind: ServiceAccount
metadata:
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::11111111:role/test-runner-role
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","kind":"ServiceAccount","metadata":{"annotations":{},"name":"gitlab-runner-service-account","namespace":"gitlab-managed-apps"}}
creationTimestamp: "2021-07-29T02:16:04Z"
name: gitlab-runner-service-account
namespace: gitlab-managed-apps
resourceVersion: "2429501"
uid: cd76b72f-d86e-456c-a32b
secrets:
- name: gitlab-runner-service-account-token-asdf
config.toml contents
rbac:
create: false
serviceAccountName: gitlab-runner-service-account
clusterWideAccess: false
serviceAccountAnnotations :
eks.amazonaws.com/role-arn: "arn:aws:iam::11111111111:role/test-runner-role"
runners:
runUntagged: true
config: |
[[runners]]
[runners.kubernetes]
namespace = "{{.Release.Namespace}}"
image = "alpine/git"
memory_limit = "7Gi"
[runners.cache]
Type = "s3"
Path = "path"
Shared = true
[runners.cache.s3]
ServerAddress = "s3.amazonaws.com"
BucketName = "gitlab-runner-cache"
BucketLocation = "us-east-2"
serviceAccountName: "gitlab-runner-service-account"
Used GitLab Runner version
Possible fixes
Edited by Tim Fendt