Webservice deployment ignores configured setting for service-account value (forcing default value: 'gitlab-app-anyuid') => Breaks AWS S3 object store usage
Summary
GitLab Operator IAM Role / service-account definition set as described in documenentation:
- https://docs.gitlab.com/charts/advanced/external-object-storage/aws-iam-roles.html#using-pre-created-service-accounts
- https://docs.gitlab.com/charts/advanced/external-object-storage/aws-iam-roles.html#using-chart-owned-service-accounts
is overwritten for webservice(workhorse) deployment by default value: 'gitlab-app-anyuid'.
We discovered it's cause by analysing source-code:
it was merged here: !447 (diffs)
In our usecase we wanted to use S3 object store. By following the documentation for helm chart mentioned above we would assume that we can affect creation of k8s serviceaccount by fields:
apiVersion: v1
kind: ServiceAccount
metadata:
name: some-serviceaccount
...MORE VALUES...
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::123456789:role/SomeRoleRole-1234567890987654321
or use some service account that was created by us as mentioned here: https://docs.gitlab.com/charts/advanced/external-object-storage/aws-iam-roles.html#using-pre-created-service-accounts
gitlab:
webservice:
serviceAccount:
create: false
name: some-serviceaccount
however operator overwrites them with it's own - 'gitlab-app-anyuid'
The only workaround we found is manually adding annotation to ServiceAccount gitlab-app-anyuid after applying default operator manifest - but that would mean all of the deployments that uses this gitlab-app-anyuid would automatically get access to AWS role which would break our security guidelines.
If we don't set a service account with the proper IAM Role webservice fails uploading artifacts to object storage with a HTTP 403:
Completed 422 Unprocessable Entity in 0ms (ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 236)
Excon::Error::Forbidden (Expected(200) <=> Actual(403 Forbidden)
excon.error.response
:body=> ""
:cookies=> [
]
:headers=> {
"Content-Type"=> "application/xml"
"Date"=> "Wed, 02 Nov 2022 13:35:56 GMT"
"Server"=> "AmazonS3"
"x-amz-id-2"=> "fhefh2848647z/yfdh27g76g27bz37gr372/F3H7pbuH1H1M="
"x-amz-request-id" => "FC5XYFTMJTXNRYZH"
}
:host=> "xyz-gitlab-artifacts-storage.s3.eu-central-1.amazonaws.com"
:local_address => "172.16.1.101"
:local_port=> 42392
:path=> "/tmp/uploads/1667396156-35-0002-6385-169d5ec5fd7f6e2e0c4ace53287a45f1"
:port=> 443
:reason_phrase => "Forbidden"
:remote_ip=> "1.2.3.4"
:status=> 403
:status_line=> "HTTP/1.1 403 Forbidden\r\n"
):
Steps to reproduce
In order to use AWS S3 object storage we need to set a own IAM Role and follow the documentation mentioned above by setting:
gitlab:
# skipped values
webservice:
serviceAccount:
create: false
enabled: true
name: s3-storage-objects-sa
but in the webservice/workhorse the default service account 'gitlab-app-anyuid' is set.
Configuration used
Relevant part of our chart with sanitized values:
apiVersion: apps.gitlab.com/v1beta1
kind: GitLab
metadata:
name: gitlab
namespace: gitlab-system
spec:
chart:
# version needs to match currently deployed operator, see: https://gitlab.com/gitlab-org/cloud-native/gitlab-operator/-/blob/master/CHART_VERSIONS
version: 6.5.1
values:
gitlab:
# skipped values
webservice:
serviceAccount:
create: false
enabled: true
name: s3-storage-objects-sa
# skipped values
global:
edition: ce
nodeSelector:
gitlab_system: "true"
tolerations:
- effect: NoSchedule
key: gitlab_system
operator: Equal
value: "true"
hosts:
# main domain root - sanitized:
domain: gitlab-poc.example.com
https: true
ingress:
apiVersion: networking.k8s.io/v1
enabled: true
configureCertmanager: false
tls:
enabled: false
shell:
tcp:
proxyProtocol: true
minio:
enabled: false
appConfig:
object_store:
enabled: true
proxy_download: true
connection:
secret: gitlab-rails-storage
key: connection
lfs:
enabled: true
bucket: lfs-gitlab-artifacts-storage
proxy_download: true
artifacts:
enabled: true
proxy_download: true
# direct_upload: true
bucket: artifacts-gitlab-artifacts-storage
uploads:
enabled: true
# direct_upload: true
proxy_download: true
bucket: uploads-gitlab-artifacts-storage
# skipped values
Current behavior
regardless to what we set in the helm-chart service-account 'gitlab-app-anyuid' is always set for the webservice pod.
Expected behavior
webservice pod service-account is using service account provided via helm chart (as described in helm chart documentation).
Versions
- Operator: 6.5.1
- Platform:
- Cloud: AWS
- Kubernetes:
- Client: 1.23.3
- Server: 1.23.10-eks-15b7512