feat: Enable automatic DB reindexing using CronJob
What does this MR do?
Related to gitlab-com/gl-infra/delivery#21668 (closed)
Automatic database reindexing is possible in an Omnibus installation by flipping some
configuration values inside the gitlab.rb file. This is not possible in the Helm chart installation,
which uses CNG (Cloud Native GitLab) container images. There is no native method for runnig
reindexing periodically, and users have to resort to running this manually in the Toolbox pod or
install their own CronJob.
The Toolbox chart already has a CronJob for taking a backup periodically. In line with that pattern, this patch adds a CronJob that will perform Automatic database reindexing on a user-specified schedule. The CronJob can be installed using these values:
gitlab:
toolbox:
enabled: true
databaseReindex:
cron:
enabled: true
schedule: "12 * * * 0,6"The script used in this MR was added to CNG in gitlab-org/build/CNG!2712 (merged) Before running the script, we ensure that its dependencies (Postgres and Redis) are accessible.
The underlying script will run a Rake task, which emulates the behavior from Omnibus, when the configuration flags mentioned here are enabled.
The database reindexing CronJob does not need variables related to persistence or backup. Furthermore, most environment variables which are not required have also been removed. The GitLab Geo team have confirmed that the reindexing does not need to run on the secondary site, as the Geo tracking database is excluded from the reindexing Rake task.
Changelog: added
What Does gitlab:db:reindex Do?
This Rake task has limitations on the job's workload that are defined inside the GitLab codebase: It will create at most 2 indexes (asynchronously) and re-index at most 2 existing indexes. So, it will not run indefinitely, and we don't need to impose any other limits on it. The code identifies what to re-index using an index bloat heuristic.
-
db.rake-
Async index creation
- At most 2 indexes will be created
-
Async constraint validation
- At most 2 constraints will be validated
-
Reindexing
- At most 2 existing indexes will be reindexed
-
Async index creation
Resource requests
Resource requests were decided based on testing done by Clemens in a GKE installation. GKE's UI shows the following usage of CPU and memory for the Pod created by this Job:
Testing notes
I tested this change by installing GitLab using the Helm chart in a Kubernetes cluster in GKE, containing 6 n2-standard-4 nodes.
When gitlab.toolbox.databaseReindex.cron.enabled = true, I confirmed the following things in this environment
-
Is the CronJob manifest generated correctly?
-
Yes.
CronJob manifest
--- # Source: gitlab/charts/gitlab/charts/toolbox/templates/database-reindex-job.yaml apiVersion: batch/v1 kind: CronJob metadata: name: gitlab-helm-chart-toolbox-db-reindex namespace: default labels: app: toolbox chart: toolbox-9.7.0 release: gitlab-helm-chart heritage: Helm spec: concurrencyPolicy: Replace failedJobsHistoryLimit: 1 schedule: "12 * * * 0,6" startingDeadlineSeconds: successfulJobsHistoryLimit: 3 suspend: false jobTemplate: spec: backoffLimit: 6 template: metadata: labels: app: toolbox chart: toolbox-9.7.0 release: gitlab-helm-chart heritage: Helm annotations: checksum/config: 16d18b58d279fd9cbb2e98ebb081ddd22fc9c1727baf2fdd6b445d4dd3f97202 cluster-autoscaler.kubernetes.io/safe-to-evict: "false" spec: restartPolicy: OnFailure securityContext: runAsUser: 1000 runAsGroup: 1000 fsGroup: 1000 seccompProfile: type: RuntimeDefault initContainers: - name: certificates image: registry.gitlab.com/gitlab-org/build/cng/certificates:v18.7.0 securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL runAsNonRoot: true runAsUser: 1000 env: - name: TZ value: "UTC" volumeMounts: - name: etc-ssl-certs mountPath: /etc/ssl/certs readOnly: false - name: etc-pki-ca-trust-extracted-pem mountPath: /etc/pki/ca-trust/extracted/pem readOnly: false resources: requests: cpu: 50m - name: configure command: ['sh', '/config/configure'] image: "registry.gitlab.com/gitlab-org/build/cng/gitlab-base:v18.7.0" securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL runAsNonRoot: true runAsUser: 1000 env: - name: TZ value: "UTC" volumeMounts: - name: toolbox-config mountPath: /config readOnly: true - name: init-toolbox-secrets mountPath: /init-config readOnly: true - name: toolbox-secrets mountPath: /init-secrets readOnly: false resources: requests: cpu: 50m containers: - name: toolbox-db-reindex args: - /bin/bash - -c - /scripts/wait-for-deps /scripts/db-reindex image: "registry.gitlab.com/gitlab-org/build/cng/gitlab-toolbox-ee:master" securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL runAsNonRoot: true runAsUser: 1000 env: - name: CONFIG_TEMPLATE_DIRECTORY value: '/var/opt/gitlab/templates' - name: CONFIG_DIRECTORY value: '/srv/gitlab/config' - name: BYPASS_SCHEMA_VERSION value: 'true' - name: TZ value: "UTC" volumeMounts: - name: toolbox-config mountPath: '/var/opt/gitlab/templates' - name: toolbox-secrets mountPath: '/etc/gitlab' readOnly: true - name: toolbox-secrets mountPath: /srv/gitlab/config/secrets.yml subPath: rails-secrets/secrets.yml - name: toolbox-tmp mountPath: '/srv/gitlab/tmp' readOnly: false - name: etc-ssl-certs mountPath: /etc/ssl/certs/ readOnly: true - name: etc-pki-ca-trust-extracted-pem mountPath: /etc/pki/ca-trust/extracted/pem readOnly: true resources: requests: cpu: 50m memory: 350M volumes: - name: toolbox-config projected: sources: - configMap: name: gitlab-helm-chart-toolbox - name: toolbox-tmp emptyDir: {} - name: init-toolbox-secrets projected: defaultMode: 0400 sources: - secret: name: "gitlab-helm-chart-rails-secret" items: - key: secrets.yml path: rails-secrets/secrets.yml - secret: name: "gitlab-helm-chart-gitlab-shell-secret" items: - key: "secret" path: shell/.gitlab_shell_secret - secret: name: "gitlab-helm-chart-gitaly-secret" items: - key: "token" path: gitaly/gitaly_token - secret: name: "gitlab-helm-chart-redis-secret" items: - key: "secret" path: redis/redis-password - secret: name: "gitlab-helm-chart-postgresql-password" items: - key: "postgresql-password" path: postgres/psql-password-ci - secret: name: "gitlab-helm-chart-postgresql-password" items: - key: "postgresql-password" path: postgres/psql-password-main - secret: name: "gitlab-helm-chart-registry-secret" items: - key: registry-auth.key path: registry/gitlab-registry.key - secret: name: "gitlab-helm-chart-registry-notification" items: - key: "secret" path: registry/notificationSecret # mount secret for minio - secret: name: "gitlab-helm-chart-minio-secret" items: - key: accesskey path: minio/accesskey - key: secretkey path: minio/secretkey # mount secret for object_store # mount secret for artifacts # mount secret for lfs # mount secret for uploads # mount secret for packages # mount secret for external_diffs # mount secret for terraform_state # mount secret for ci_secure_files # mount secret for dependency_proxy # mount secret for pages # mount secrets for LDAP # mount secrets for microsoftGraphMailer - name: toolbox-secrets emptyDir: medium: "Memory" - name: etc-ssl-certs emptyDir: medium: "Memory" - name: etc-pki-ca-trust-extracted-pem emptyDir: medium: "Memory"
-
-
Does the CronJob get installed correctly?
- Yes.
$ kubectl 2>/dev/null get --no-headers cronjob gitlab-helm-chart-toolbox-db-reindex 12 * * * 0,6 <none> False 0 <none> 3m26s
- Yes.
-
Does the CronJob run correctly?
- Yes. But no table was re-indexed because this is a small instance without any data.
$ kubectl 2>/dev/null get --no-headers job | grep --color=never reindex db-reindex-testing-timestamp-1767150267 Complete 1/1 43s 76s $ k logs -f db-reindex-testing-timestamp-1767150267-7thbm Defaulted container "toolbox-db-reindex" out of: toolbox-db-reindex, certificates (init), configure (init) Begin parsing .erb templates from /var/opt/gitlab/templates Writing /srv/gitlab/config/cable.yml Writing /srv/gitlab/config/database.yml Writing /srv/gitlab/config/gitlab.yml Writing /srv/gitlab/config/resque.yml Writing /srv/gitlab/config/session_store.yml Begin parsing .tpl templates from /var/opt/gitlab/templates Copying other config files found in /var/opt/gitlab/templates to /srv/gitlab/config Copying smtp_settings.rb into /srv/gitlab/config Attempting to run '/bin/bash -c /scripts/wait-for-deps /scripts/db-reindex' as a main process Performing automatic database reindexing
- Yes. But no table was re-indexed because this is a small instance without any data.
When gitlab.toolbox.databaseReindex.cron.enabled = false, the CronJob is not installed. If the CronJob is already installed, it is not removed. So, the user has to remove it manually if they want to stop using the CronJob after testing it once.
Testing environment and values
I used the values file examples/values-gke-minimum.yaml and appended some over-rides on top of it:
Individual overrides files
$ cat values-required.yaml
certmanager-issuer:
email: abcd@example.com
$ cat values-overrides.yaml
global:
gitlabVersion: 18.7.0
hosts:
domain: example.com
# nginx-ingress-controller service's EXTERNAL-IP
externalIP: 0.0.0.0
kas:
enabled: false
registry:
enabled: false
registry:
enabled: false
gitlab:
migrations:
enabled: true
toolbox:
enabled: true
webservice:
minReplicas: 1
resources:
limits:
memory: 2.5G
requests:
cpu: 100m
memory: 2G
# Reduce the Puma worker processes to 0
# This installation is low throughput.
workerProcesses: 0
# I want to disable the PDB but I can not do that. So, I will just increase the maxUnavailable
# value.
maxUnavailable: 5
$ cat values-enable-database-reindex.yaml
gitlab:
toolbox:
image:
# We need https://gitlab.com/gitlab-org/build/CNG/-/merge_requests/2712 which has not been released yet.
tag: master
databaseReindex:
cron:
enabled: true
concurrencyPolicy: Replace
failedJobsHistoryLimit: 1
schedule: "12 * * * 0,6"
startingDeadlineSeconds: null
successfulJobsHistoryLimit: 3
suspend: false
backoffLimit: 6
# activeDeadlineSeconds:
# ttlSecondsAfterFinished:
safeToEvict: false
restartPolicy: "OnFailure"
resources:
# limits:
# cpu: 1
# memory: 2G
requests:
cpu: 50m
memory: 350M
~/code/gitlab-org/charts/gitlab $ The complete set of user-supplied values were:
Complete set of user-supplied values reported by Helm
$ helm upgrade --install --values values-gke-minimum.yaml --values values-required.yaml --values values-overrides.yaml --values values-enable-database-reindex.yaml --debug --wait-for-jobs --dry-run gitlab-helm-chart ./
history.go:56: 2025-12-31 14:44:58.309701845 +0900 JST m=+0.555331582 [debug] getting history for release gitlab-helm-chart
upgrade.go:164: 2025-12-31 14:45:00.614385979 +0900 JST m=+2.860015710 [debug] preparing upgrade for gitlab-helm-chart
upgrade.go:172: 2025-12-31 14:45:04.817969076 +0900 JST m=+7.063598801 [debug] performing update for gitlab-helm-chart
upgrade.go:366: 2025-12-31 14:45:08.513512 +0900 JST m=+10.759141737 [debug] dry run for gitlab-helm-chart
Release "gitlab-helm-chart" has been upgraded. Happy Helming!
NAME: gitlab-helm-chart
LAST DEPLOYED: Wed Dec 31 14:45:02 2025
NAMESPACE: default
STATUS: pending-upgrade
REVISION: 6
USER-SUPPLIED VALUES:
certmanager:
resources:
requests:
cpu: 10m
memory: 32Mi
certmanager-issuer:
email: abcd@example.com
gitlab:
gitlab-shell:
minReplicas: 1
migrations:
enabled: true
sidekiq:
minReplicas: 1
resources:
limits:
memory: 1.5G
requests:
cpu: 50m
memory: 625M
toolbox:
databaseReindex:
cron:
backoffLimit: 6
concurrencyPolicy: Replace
enabled: true
failedJobsHistoryLimit: 1
resources:
requests:
cpu: 50m
memory: 350M
restartPolicy: OnFailure
safeToEvict: false
schedule: 12 * * * 0,6
startingDeadlineSeconds: null
successfulJobsHistoryLimit: 3
suspend: false
enabled: true
image:
tag: master
webservice:
maxUnavailable: 5
minReplicas: 1
resources:
limits:
memory: 2.5G
requests:
cpu: 100m
memory: 2G
workerProcesses: 0
workhorse:
resources:
limits:
memory: 100M
requests:
cpu: 10m
memory: 10M
gitlab-runner:
install: false
global:
gitlabVersion: 18.7.0
hosts:
domain: example.com
externalIP: 0.0.0.0
kas:
enabled: false
rails:
bootsnap:
enabled: false
registry:
enabled: false
minio:
resources:
requests:
cpu: 10m
memory: 64Mi
nginx-ingress:
controller:
minAvailable: 0
replicaCount: 1
resources:
requests:
cpu: 50m
memory: 100Mi
prometheus:
install: false
redis:
resources:
requests:
cpu: 10m
memory: 64Mi
registry:
enabled: false
hpa:
minReplicas: 1
Related issues
Author checklist
For general guidance, please follow our Contributing guide.
Required
For anything in this list which will not be completed, please provide a reason in the MR discussion.
- Merge Request Title and Description are up to date, accurate, and descriptive.
- MR targeting the appropriate branch.
- MR has a green pipeline.
- Documentation created/updated.
- Tests added/updated, and test plan for scenarios not covered by automated tests.
-
Equivalent MR/issue for omnibus-gitlab opened.
- Not required. Omnibus already has this feature. See https://docs.gitlab.com/omnibus/settings/database/#automatic-database-reindexing for the configuration flags which should be set to use this.
Reviewers checklist
- MR has a green pipeline on https://gitlab.com/gitlab-org/charts/gitlab.
- Consider downstream impact to the Operator, as per evaluating impact from changes to GitLab chart.