S3 Object Storages gets error 500 and "Access Denied" in logs of dockerized Gitlab configured with IAM
Summary
The all object storage doesn't working our self-hosted (omnibus docker install) GitLab server versions: 14.10.2-ce.0, 15.6.1-ce.0 and 15.5.6-ce.0(latest).
Our object storages are set up to use an AWS S3 bucket. After trying to change a avatar in user profile or define artifacts in CI, we get a 500 error in Gitlab WEB UI.
We have verified that the AWS EKS pod running GitLab has a role with a policy that allows the necessary Get/Put actions on the S3 bucket.
# kubectl exec -it gitlab-srv-0 bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
root@gitlab-srv-0:/# curl --silent --location "https://awscli.amazonaws.com/awscli-exe-linux-x86_64-2.4.27.zip" -o "awscliv2.zip" && unzip awscliv2.zip && ./aws/install
root@gitlab-srv-0:/# aws s3 ls
2022-12-14 23:22:26 <artifacts-bucket-name>
2022-12-14 23:22:26 <backup-bucket-name>
2022-12-14 23:22:24 <lfs-bucket-name>
2022-12-14 23:22:24 <packages-bucket-name>
2022-09-22 11:49:23 <tf-state-bucket-name>
2022-12-14 23:22:26 <uploads-bucket-name>
root@gitlab-srv-0:/# aws s3 cp /etc/gitlab/gitlab.rb s3://<upload-bucket-name>/1/test.txt
upload: etc/gitlab/gitlab.rb to s3://<upload-bucket-name>/1/test.txt
root@gitlab-srv-0:/# Steps to reproduce
- Start docker container
gitlab/gitlab-ce:14.10.2-ce.0(this also relevant to gitlab/gitlab-ce:15.7.0-ce.0) in EKS with environment variableGITLAB_OMNIBUS_CONFIGhas this value:
value: |
external_url '<GITLAB URL>'
nginx['listen_https'] = false
nginx['listen_port'] = 80
letsencrypt['enable'] = false
git_data_dir '/gitlab-data/git-data'
gitlab_rails['shared_path'] = '/gitlab-data/shared'
gitlab_ci['builds_directory'] = '/gitlab-data/builds'
gitlab_rails['object_store']['enabled'] = true
gitlab_rails['object_store']['proxy_download'] = false
gitlab_rails['object_store']['connection'] = {
'provider' => 'AWS',
'region' => '{{ .Values.region }}',
'use_iam_profile' => true
}
gitlab_rails['object_store']['objects']['artifacts']['bucket'] = '<artifacts-bucket-name>'
gitlab_rails['object_store']['objects']['uploads']['enabled'] = true
gitlab_rails['object_store']['objects']['uploads']['bucket'] = '<uploads-bucket-name>'
gitlab_rails['object_store']['objects']['lfs']['bucket'] = '<lfs-bucket-name>'
gitlab_rails['object_store']['objects']['packages']['enabled'] = false
gitlab_rails['object_store']['objects']['terraform_state']['enabled'] = false
gitlab_rails['object_store']['objects']['dependency_proxy']['enabled'] = false
gitlab_rails['object_store']['objects']['external_diffs']['enabled'] = false
gitlab_rails['object_store']['objects']['pages']['enabled'] = false- Configure trusted IAM role in AWS
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::<accont>:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/<EKS ID>"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"oidc.eks.us-east-1.amazonaws.com/id/<EKS ID>:sub": "system:serviceaccount:<gitlab namespace>:<name of SA>",
"oidc.eks.us-east-1.amazonaws.com/id/<EKS ID>:aud": "sts.amazonaws.com"
}
}
}
]
}- Configure
S3 policy
{
"Statement": [
{
"Action": "s3:*",
"Effect": "Allow",
"Resource": "arn:aws:s3:::*",
"Sid": ""
},
{
"Action": "s3:*",
"Effect": "Allow",
"Resource": "arn:aws:s3:::*/*",
"Sid": ""
}
],
"Version": "2012-10-17"
}
- Try to change avatar in profile of root user
- Problem persists
What is the current bug behavior?
Changing avatars or pushing artifacts from CI results error message:
Request failed with status code 500What is the expected correct behavior?
Changing avatars or pushing artifacts from CI are uploaded and stored in the configured AWS S3 bucket.
Relevant logs and/or screenshots
The relevant lines from production.log seem to be:
Excon::Error::Forbidden (Expected(200) <=> Actual(403 Forbidden)
excon.error.response
:body => "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>5N6QE2Z1916JMR1A</RequestId><HostId>QPsPFVurDzpGtT4PCUN37pDAY4i/EJwCmWj7cftT/lIIwLxsZRwcVyssQeFn82H3LIv0BmhDbhI=</HostId></Error>"
:cookies => [
]
:headers => {
"Connection" => "close"
"Content-Type" => "application/xml"
"Date" => "Thu, 15 Dec 2022 00:20:00 GMT"
"Server" => "AmazonS3"
"x-amz-id-2" => "<id>"
"x-amz-request-id" => "5N6QE2Z1916JMR1A"
}
:host => "<upload-bucket-name>.s3.amazonaws.com"
:local_address => "10.27.245.69"
:local_port => 42458
:path => "/user/avatar/1/avatar.png"
:port => 443
:reason_phrase => "Forbidden"
:remote_ip => "54.231.171.73"
:status => 403
:status_line => "HTTP/1.1 403 Forbidden\r\n"
):
app/uploaders/object_storage.rb:379:in `store!'Output of checks
Checks with gitlab-rails console
ENV['EXCON_DEBUG'] = 'true'
#connection = Fog::Storage.new(Gitlab.config.object_store.connection.deep_symbolize_keys)
:body => "<AssumeRoleWithWebIdentityResponse xmlns=\"https://sts.amazonaws.com/doc/2011-06-15/\"
<AssumeRoleWithWebIdentityResult>
<Audience>sts.amazonaws.com</Audience>
<AssumedRoleUser>
<AssumedRoleId>AROATJNT3IK36GRU4MHAS:fog-aws-ad64f62b302b2256d99505df2b97e8aa</AssumedRoleId>
| <Arn>arn:aws:sts::<account>:assumed-role/<correct s3 role from reproduced step>/fog-aws-ad9505df2b97e8aa</Arn>
</AssumedRoleUser>
:headers => {
"Content-Length" => "2388"
"Content-Type" => "text/xml"
"Date" => "Thu, 15 Dec 2022 00:23:22 GMT"
"x-amzn-RequestId" => "34cd4981-732a-434e-acaa-3a4b4310391e"
}
:host => "sts.us-east-1.amazonaws.com"
:local_address => "10.27.245.69"
:local_port => 51618
:path => "/"
:port => 443
:reason_phrase => "OK"
:remote_ip => "10.27.247.31"
:status => 200
:status_line => "HTTP/1.1 200 OK\r\n"
--
#connection.initiate_multipart_upload('<uploads-bucke-name>', 'test.txt')
excon.request
:chunk_size => 1048576
:ciphers => "ECDHE-ECDSA-CHACHA20-POLY1305:...."
:connect_timeout => 60
:debug_request => true
:debug_response => true
:expects => 200
:headers => {
"Authorization" => "REDACTED"
"Host" => "<uploads-bucke-name>.s3.amazonaws.com"
"User-Agent" => "fog-core/2.1.0"
"x-amz-content-sha256" => "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
"x-amz-date" => "20221215T002334Z"
"x-amz-security-token" => "IQo..."
:host => "<uploads-bucke-name>.s3.amazonaws.com"
:hostname => "<uploads-bucke-name>.s3.amazonaws.com"
:idempotent => false
:instrumentor => Excon::StandardInstrumentor
:instrumentor_name => "excon"
:method => "POST"
:omit_default_port => false
:path => "/test.txt"
:persistent => false
:port => 443
:query => {
"uploads" => nil
`excon.response`
:body => ""
:headers => {
"Date" => "Thu, 15 Dec 2022 00:23:35 GMT"
"Server" => "AmazonS3"
"x-amz-id-2" => "qtOZ4GQab4QwuVB3n3iRzZESpNG9lobxlYwcKdttnlU6o7CCv84JQ8o0E2w84T2NI="
"x-amz-request-id" => "4V5NC5XYYK5E4S7J"
"x-amz-server-side-encryption" => "AES256"
}
:host => "<uploads-bucke-name>.s3.amazonaws.com"
:local_address => "10.27.245.69"
:local_port => 44820
:path => "/test.txt"
:port => 443
:reason_phrase => "OK"
:remote_ip => "52.217.140.241"
:status => 200
:status_line => "HTTP/1.1 200 OK"Results of GitLab environment info
System information System: Current User: git Using RVM: no Ruby Version: 2.7.5p203 Gem Version: 3.1.4 Bundler Version:2.2.33 Rake Version: 13.0.6 Redis Version: 6.2.6 Sidekiq Version:6.4.0 Go Version: unknown
GitLab information Version: 14.10.2 Revision: 07d12f3fd11 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 13.8 URL: http://localhost:8280 HTTP Clone URL: http://localhost:8280/some-group/some-project.git SSH Clone URL: git@localhost:some-group/some-project.git Using LDAP: no Using Omniauth: yes Omniauth Providers:
GitLab Shell Version: 13.25.1 Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell
Edited by Nikolai Novikov