database-upgrade "post" fails to restore replicas count if multiple deployments
Summary
When ugprading releases with the chart-provided postgresql enabled to GitLab 18.0 (chart 9.0.x, from 8.11.x), following the upgrade doc, the database-upgrade script fails at the end of the post step, while restoring replicas counts on deployments, if there are several deployments for some apps (multiple webservice in my case, would be the same for multiple sidekiq).
That's not a big deal, it's easy to fix with some kubectl scale commands to finish the process, but I'll send an MR to save the trouble for others.
Steps to reproduce
With a deployments setup like this (multiple webservice deployments):
$ kubectl get deployments.apps
NAME READY UP-TO-DATE AVAILABLE AGE
gitlab-dv-gitlab-exporter 1/1 1 1 488d
gitlab-dv-gitlab-pages 1/1 1 1 488d
gitlab-dv-gitlab-shell 1/1 1 1 488d
gitlab-dv-minio 1/1 1 1 488d
gitlab-dv-nginx-ingress-controller 2/2 2 2 214d
gitlab-dv-registry 1/1 1 1 488d
gitlab-dv-sidekiq-catchall-v2 1/1 1 1 488d
gitlab-dv-toolbox 1/1 1 1 488d
gitlab-dv-webservice-api 1/1 1 1 488d
gitlab-dv-webservice-default 1/1 1 1 488d
gitlab-dv-webservice-internal 1/1 1 1 488dStarting on a chart 8.11 setup with postgresql enabled, and following the upgrade steps to migrate to pg16.
Running the post step of database-upgrade starts and ends with these messages:
$ /tmp/database-upgrade post
configmap/database-upgrade created
configmap/database-upgrade patched
Stopping: gitlab-dv-webservice-api
deployment.apps/gitlab-dv-webservice-api patched (no change)
configmap/database-upgrade patched
Stopping: gitlab-dv-sidekiq-catchall-v2
deployment.apps/gitlab-dv-sidekiq-catchall-v2 patched (no change)
configmap/database-upgrade patched
Stopping: gitlab-dv-gitlab-exporter
deployment.apps/gitlab-dv-gitlab-exporter patched
configmap/database-upgrade patched
Stopping: gitlab-dv-registry
deployment.apps/gitlab-dv-registry patched
Unpacking backup
[...]
main: == 20250519155241 AddNotNullOrganizationIdOnForkNetworks: migrating ===========
main: -- change_column_null(:fork_networks, :organization_id, false, nil)
main: -> 0.0042s
main: == 20250519155241 AddNotNullOrganizationIdOnForkNetworks: migrated (0.0138s) ==
main: == [advisory_lock_connection] object_id: 47960, pg_backend_pid: 767
Starting: gitlab-dv-sidekiq-catchall-v2
deployment.apps/gitlab-dv-sidekiq-catchall-v2 patched (no change)
Starting: gitlab-dv-webservice-api
Error from server (BadRequest): invalid character 'g' after object key:value pairNote that only one gitlab-dv-webservice-... deployment is mentioned, and that at the end restoring the replicas count fails and the script exits.
Proposed fix
The database-upgrade configmap, where replicas counts are stored, gives a good hint for understanding the issue:
$ kubectl get configmaps database-upgrade -o yaml
apiVersion: v1
data:
gitlab-dv-gitlab-exporter: "1"
gitlab-dv-registry: "1"
gitlab-dv-sidekiq-catchall-v2: "0"
gitlab-dv-webservice-api: 0gitlab-dv-webservice-default
kind: ConfigMap
metadata:
creationTimestamp: "2025-06-23T08:49:24Z"
name: database-upgrade
namespace: gitlab-dv
resourceVersion: "449996341"
uid: 8d44423f-f99b-47bb-b5f9-53def07bfd1dHere is how deployments names and replicas counts are retrieved in the script:
info=$(kubectl ${namespace} get deployment -l "$(selector ${app})" -o template --template='{{ range .items }}{{.metadata.name}},{{.spec.replicas}}{{ end }}')Tested in my setup:
$ selector() {
> base="app=${1}"
> if [ -n "${release}" ]
> then
> base+=",release=${release}"
> fi
> echo "${base}"
> }
$ app=webservice
$ kubectl ${namespace} get deployment -l "$(selector ${app})" -o template --template='{{ range .items }}{{.metadata.name}},{{.spec.replicas}}{{ end }}'
gitlab-dv-webservice-api,1gitlab-dv-webservice-default,1gitlab-dv-webservice-internal,1I will submit a fix MR to output one deployment,count tuple per line there, and iterate on the results instead of assuming that there is only one or none.