Fix toolbox backup with GKE Workload Identity Federation
Summary
We are running Gitlab in production with toolbox backup pushing archives to gcs buckets. In the past we were patching the chart to create the .boto file to allow the cron to use the default credential linked to the kubernetes service account. Since chart version 8.0.0+ we saw that a fix was implemented but there is still one minor modification to add to fix incorrect projected secret init-toolbox-secrets for objectstorage config.
Steps to reproduce
in values only specify objectStorage.backend: gcs
the resulting template in toolbox backup cronjob is the following :
apiVersion: batch/v1beta1
kind: CronJob
metadata:
name: gitlab-toolbox-backup
namespace: gitlab
labels:
app: toolbox
chart: toolbox-8.2.0
release: gitlab
heritage: Helm
spec:
concurrencyPolicy: Replace
failedJobsHistoryLimit: 1
schedule: "0 1 * * *"
startingDeadlineSeconds:
successfulJobsHistoryLimit: 3
suspend: false
jobTemplate:
spec:
backoffLimit: 6
template:
metadata:
labels:
app: toolbox
chart: toolbox-8.2.0
release: gitlab
heritage: Helm
annotations:
checksum/config: 62d7612e4f622a638225b6dc64e492fda98648e74b8aa623f0a1dbee25e7bc61
cluster-autoscaler.kubernetes.io/safe-to-evict: "false"
spec:
restartPolicy: OnFailure
securityContext:
runAsUser: 1000
runAsGroup: 1000
fsGroup: 1000
serviceAccountName: gitlab
initContainers:
- name: certificates
image: registry.gitlab.com/gitlab-org/build/cng/certificates:v17.2.0
env:
volumeMounts:
- name: etc-ssl-certs
mountPath: /etc/ssl/certs
readOnly: false
- name: etc-pki-ca-trust-extracted-pem
mountPath: /etc/pki/ca-trust/extracted/pem
readOnly: false
- name: custom-ca-certificates
mountPath: /usr/local/share/ca-certificates
readOnly: true
resources:
limits:
cpu: 200m
requests:
cpu: 200m
- name: configure
command: ['sh', '/config/configure']
image: "registry.gitlab.com/gitlab-org/build/cng/gitlab-base:v17.2.0"
env:
volumeMounts:
- name: toolbox-config
mountPath: /config
readOnly: true
- name: init-toolbox-secrets
mountPath: /init-config
readOnly: true
- name: toolbox-secrets
mountPath: /init-secrets
readOnly: false
resources:
limits:
cpu: 200m
requests:
cpu: 200m
containers:
- name: toolbox-backup
args:
- /bin/bash
- -c
- sh /var/opt/gitlab/templates/configure-gsutil && backup-utility --skip artifacts,uploads
image: "registry.gitlab.com/gitlab-org/build/cng/gitlab-toolbox-ee:v17.2.0"
securityContext:
runAsUser: 1000
env:
- name: ARTIFACTS_BUCKET_NAME
value: gitlab-artifacts
- name: REGISTRY_BUCKET_NAME
value: registry
- name: LFS_BUCKET_NAME
value: git-lfs
- name: UPLOADS_BUCKET_NAME
value: gitlab-uploads
- name: PACKAGES_BUCKET_NAME
value: gitlab-packages
- name: EXTERNAL_DIFFS_BUCKET_NAME
value: gitlab-mr-diffs
- name: TERRAFORM_STATE_BUCKET_NAME
value: gitlab-terraform-state
- name: CI_SECURE_FILES_BUCKET_NAME
value: gitlab-ci-secure-files
- name: BACKUP_BUCKET_NAME
value: gitlab-backups
- name: BACKUP_BACKEND
value: gcs
- name: TMP_BUCKET_NAME
value: tmp
- name: PAGES_BUCKET_NAME
value: gitlab-pages
- name: GITALY_FEATURE_DEFAULT_ON
value: "1"
- name: CONFIG_TEMPLATE_DIRECTORY
value: '/var/opt/gitlab/templates'
- name: CONFIG_DIRECTORY
value: '/srv/gitlab/config'
- name: GOOGLE_APPLICATION_CREDENTIALS
value: '/etc/gitlab/objectstorage/config'
volumeMounts:
- name: toolbox-config
mountPath: '/var/opt/gitlab/templates'
- name: toolbox-secrets
mountPath: '/etc/gitlab'
readOnly: true
- name: toolbox-secrets
mountPath: /srv/gitlab/config/secrets.yml
subPath: rails-secrets/secrets.yml
- name: toolbox-tmp
mountPath: '/srv/gitlab/tmp'
readOnly: false
- name: etc-ssl-certs
mountPath: /etc/ssl/certs/
readOnly: true
- name: etc-pki-ca-trust-extracted-pem
mountPath: /etc/pki/ca-trust/extracted/pem
readOnly: true
resources:
requests:
cpu: 2
memory: 2Gi
volumes:
- name: toolbox-config
projected:
sources:
- configMap:
name: gitlab-toolbox
- name: toolbox-tmp
persistentVolumeClaim:
claimName: gitlab-toolbox-backup-tmp
- name: init-toolbox-secrets
projected:
defaultMode: 0400
sources:
- secret:
name: "gitlab-rails-secret"
items:
- key: secrets.yml
path: rails-secrets/secrets.yml
- secret:
name: "gitlab-gitlab-shell-secret"
items:
- key: "secret"
path: shell/.gitlab_shell_secret
- secret:
name: "gitlab-gitaly-secret"
items:
- key: "token"
path: gitaly/gitaly_token
- secret:
name: "gitlab-redis-secret"
items:
- key: "secret"
path: redis/redis-password
- secret:
name: "gitlabdb"
items:
- key: "gitlab"
path: postgres/psql-password-ci
- secret:
name: "gitlabdb"
items:
- key: "gitlab"
path: postgres/psql-password-main
- secret:
name: "gitlab-registry-secret"
items:
- key: registry-auth.key
path: registry/gitlab-registry.key
- secret:
name: "gitlab-registry-notification"
items:
- key: "secret"
path: registry/notificationSecret
- secret:
name:
items:
- key: config
path: objectstorage/config
# mount secret for minio
# mount secret for object_store
- secret:
name: gitlab-storage-secret
items:
- key: connection
path: objectstorage/object_store
- secret:
name: gitlab-google-oauth2
items:
- key: provider
path: "omniauth/gitlab-google-oauth2/provider"
# mount secrets for microsoftGraphMailer
# mount secret for artifacts
# mount secret for lfs
# mount secret for uploads
# mount secret for packages
# mount secret for external_diffs
# mount secret for terraform_state
# mount secret for ci_secure_files
# mount secret for dependency_proxy
# mount secret for pages
# mount secrets for LDAP
- name: toolbox-secrets
emptyDir:
medium: "Memory"
- name: etc-ssl-certs
emptyDir:
medium: "Memory"
- name: etc-pki-ca-trust-extracted-pem
emptyDir:
medium: "Memory"
- name: custom-ca-certificates
projected:
defaultMode: 0440
sources:
- secret:
name: gitlab-wildcard-tls-caCurrent behavior
We seee the error in the rendered CronJob object
- secret:
name:
items:
- key: config
path: objectstorage/configExpected behavior
apiVersion: batch/v1beta1
kind: CronJob
metadata:
name: gitlab-toolbox-backup
namespace: gitlab
labels:
app: toolbox
chart: toolbox-8.2.0
release: gitlab
heritage: Helm
spec:
concurrencyPolicy: Replace
failedJobsHistoryLimit: 1
schedule: "0 1 * * *"
startingDeadlineSeconds:
successfulJobsHistoryLimit: 3
suspend: false
jobTemplate:
spec:
backoffLimit: 6
template:
metadata:
labels:
app: toolbox
chart: toolbox-8.2.0
release: gitlab
heritage: Helm
annotations:
checksum/config: 62d7612e4f622a638225b6dc64e492fda98648e74b8aa623f0a1dbee25e7bc61
cluster-autoscaler.kubernetes.io/safe-to-evict: "false"
spec:
restartPolicy: OnFailure
securityContext:
runAsUser: 1000
runAsGroup: 1000
fsGroup: 1000
serviceAccountName: gitlab
initContainers:
- name: certificates
image: registry.gitlab.com/gitlab-org/build/cng/certificates:v17.2.0
env:
volumeMounts:
- name: etc-ssl-certs
mountPath: /etc/ssl/certs
readOnly: false
- name: etc-pki-ca-trust-extracted-pem
mountPath: /etc/pki/ca-trust/extracted/pem
readOnly: false
- name: custom-ca-certificates
mountPath: /usr/local/share/ca-certificates
readOnly: true
resources:
limits:
cpu: 200m
requests:
cpu: 200m
- name: configure
command: ['sh', '/config/configure']
image: "registry.gitlab.com/gitlab-org/build/cng/gitlab-base:v17.2.0"
env:
volumeMounts:
- name: toolbox-config
mountPath: /config
readOnly: true
- name: init-toolbox-secrets
mountPath: /init-config
readOnly: true
- name: toolbox-secrets
mountPath: /init-secrets
readOnly: false
resources:
limits:
cpu: 200m
requests:
cpu: 200m
containers:
- name: toolbox-backup
args:
- /bin/bash
- -c
- sh /var/opt/gitlab/templates/configure-gsutil && backup-utility --skip artifacts,uploads
image: "registry.gitlab.com/gitlab-org/build/cng/gitlab-toolbox-ee:v17.2.0"
securityContext:
runAsUser: 1000
env:
- name: ARTIFACTS_BUCKET_NAME
value: gitlab-artifacts
- name: REGISTRY_BUCKET_NAME
value: registry
- name: LFS_BUCKET_NAME
value: git-lfs
- name: UPLOADS_BUCKET_NAME
value: gitlab-uploads
- name: PACKAGES_BUCKET_NAME
value: gitlab-packages
- name: EXTERNAL_DIFFS_BUCKET_NAME
value: gitlab-mr-diffs
- name: TERRAFORM_STATE_BUCKET_NAME
value: gitlab-terraform-state
- name: CI_SECURE_FILES_BUCKET_NAME
value: gitlab-ci-secure-files
- name: BACKUP_BUCKET_NAME
value: gitlab-backups
- name: BACKUP_BACKEND
value: gcs
- name: TMP_BUCKET_NAME
value: tmp
- name: PAGES_BUCKET_NAME
value: gitlab-pages
- name: GITALY_FEATURE_DEFAULT_ON
value: "1"
- name: CONFIG_TEMPLATE_DIRECTORY
value: '/var/opt/gitlab/templates'
- name: CONFIG_DIRECTORY
value: '/srv/gitlab/config'
- name: GOOGLE_APPLICATION_CREDENTIALS
value: '/etc/gitlab/objectstorage/config'
volumeMounts:
- name: toolbox-config
mountPath: '/var/opt/gitlab/templates'
- name: toolbox-secrets
mountPath: '/etc/gitlab'
readOnly: true
- name: toolbox-secrets
mountPath: /srv/gitlab/config/secrets.yml
subPath: rails-secrets/secrets.yml
- name: toolbox-tmp
mountPath: '/srv/gitlab/tmp'
readOnly: false
- name: etc-ssl-certs
mountPath: /etc/ssl/certs/
readOnly: true
- name: etc-pki-ca-trust-extracted-pem
mountPath: /etc/pki/ca-trust/extracted/pem
readOnly: true
resources:
requests:
cpu: 2
memory: 2Gi
volumes:
- name: toolbox-config
projected:
sources:
- configMap:
name: gitlab-toolbox
- name: toolbox-tmp
persistentVolumeClaim:
claimName: gitlab-toolbox-backup-tmp
- name: init-toolbox-secrets
projected:
defaultMode: 0400
sources:
- secret:
name: "gitlab-rails-secret"
items:
- key: secrets.yml
path: rails-secrets/secrets.yml
- secret:
name: "gitlab-gitlab-shell-secret"
items:
- key: "secret"
path: shell/.gitlab_shell_secret
- secret:
name: "gitlab-gitaly-secret"
items:
- key: "token"
path: gitaly/gitaly_token
- secret:
name: "gitlab-redis-secret"
items:
- key: "secret"
path: redis/redis-password
- secret:
name: "gitlabdb"
items:
- key: "gitlab"
path: postgres/psql-password-ci
- secret:
name: "gitlabdb"
items:
- key: "gitlab"
path: postgres/psql-password-main
- secret:
name: "gitlab-registry-secret"
items:
- key: registry-auth.key
path: registry/gitlab-registry.key
- secret:
name: "gitlab-registry-notification"
items:
- key: "secret"
path: registry/notificationSecret
# mount secret for minio
# mount secret for object_store
- secret:
name: gitlab-storage-secret
items:
- key: connection
path: objectstorage/object_store
- secret:
name: gitlab-google-oauth2
items:
- key: provider
path: "omniauth/gitlab-google-oauth2/provider"
# mount secrets for microsoftGraphMailer
# mount secret for artifacts
# mount secret for lfs
# mount secret for uploads
# mount secret for packages
# mount secret for external_diffs
# mount secret for terraform_state
# mount secret for ci_secure_files
# mount secret for dependency_proxy
# mount secret for pages
# mount secrets for LDAP
- name: toolbox-secrets
emptyDir:
medium: "Memory"
- name: etc-ssl-certs
emptyDir:
medium: "Memory"
- name: etc-pki-ca-trust-extracted-pem
emptyDir:
medium: "Memory"
- name: custom-ca-certificates
projected:
defaultMode: 0440
sources:
- secret:
name: gitlab-wildcard-tls-caProposed Solution
In the charts/gitlab/charts/toolbox/templates/_helpers.tpl add an if close if the toolbox.backups.objectStorage.config is not defined
{{ include "toolbox.backups.objectStorage.config.secret" .Values.backups.objectStorage }}
*/}}
{{- define "toolbox.backups.objectStorage.config.secret" -}}
{{- if eq .backend "gcs" -}}
{{- if not .config -}}
{{- else -}}
- secret:
name: {{ .config.secret }}
items:
- key: {{ default "config" .config.key }}
path: objectstorage/{{ default "config" .config.key }}
{{- end -}}
{{- else if eq .backend "azure" -}}
- secret:
name: {{ .config.secret }}
items:
- key: {{ default "config" .config.key }}
path: objectstorage/azure_config
{{- else -}}
- secret:
name: {{ .config.secret }}
items:
- key: {{ default "config" .config.key }}
path: objectstorage/.s3cfg
{{- end -}}
{{- end -}}