GitLab backup-utility failing copying tar file to GCP bucket

Summary

After upgrading our self hosted GitLab (Helm / GKE) to v17.6.2-ee, the backup-utility tool fails with the following error:

module 'OpenSSL.crypto' has no attribute 'sign'

Steps to reproduce

kubectl -n development exec -it <toolbox-pod> -- sh
backup-utility --backend gcs

Example Project

What is the current bug behavior?

The backup-utility creates the tar file but fails when trying to copy the tar file to the GCP bucket using the gsutil tool and exits with a non-zero return code.

What is the expected correct behavior?

The backup-utility tool should complete without errors and the tar file should appear in the configured bucket.

Relevant logs and/or screenshots

kubectl -n development exec -it development-toolbox-7f486f647b-lb2wb -- sh

$ backup-utility --backend gcs
2024-12-19 09:19:27 UTC -- Dumping database ... 
2024-12-19 09:19:27 UTC -- Dumping PostgreSQL database development ... 
2024-12-19 09:19:36 UTC -- [DONE]
2024-12-19 09:19:36 UTC -- Dumping database ... done
2024-12-19 09:19:36 UTC -- Deleting backup and restore PID file at [/srv/gitlab/tmp/backup_restore.pid] ... done
2024-12-19 09:20:18 UTC -- Dumping repositories ... 
2024-12-19 09:20:19 UTC -- Dumping repositories ... done
2024-12-19 09:20:19 UTC -- Deleting backup and restore PID file at [/srv/gitlab/tmp/backup_restore.pid] ... done
WARNING: Active Record does not support composite primary key.

security_findings has composite primary key. Composite primary key is ignored.
Packing up backup tar
module 'OpenSSL.crypto' has no attribute 'sign'
$ echo $?
1

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info

System information
System:		
Proxy:		no
Current User:	git
Using RVM:	no
Ruby Version:	3.2.5
Gem Version:	3.5.22
Bundler Version:2.5.22
Rake Version:	13.0.6
Redis Version:	7.0.15
Sidekiq Version:7.2.4
Go Version:	unknown

GitLab information
Version:	17.6.2-ee
Revision:	6ff444aad3a
Directory:	/srv/gitlab
DB Adapter:	PostgreSQL
DB Version:	14.13
URL:		https://gitlab.developers.cam.ac.uk
HTTP Clone URL:	https://gitlab.developers.cam.ac.uk/some-group/some-project.git
SSH Clone URL:	git@gitlab.developers.cam.ac.uk:some-group/some-project.git
Elasticsearch:	yes
Geo:		no
Using LDAP:	no
Using Omniauth:	yes
Omniauth Providers: saml, github, gitlab, google_oauth2, bitbucket

GitLab Shell
Version:	14.39.0
Repository storages:
- default: 	tcp://production-gitaly-0.production-gitaly.production.svc:8075
GitLab Shell path:		/home/git/gitlab-shell

Gitaly
- default Address: 	tcp://production-gitaly-0.production-gitaly.production.svc:8075
- default Version: 	17.6.2
- default Git Version: 	2.47.0

Results of GitLab application Check

Expand for output related to the GitLab application check

(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true)
(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true)
(we will only investigate if the tests are passing)

Possible fixes

The issue is caused by gsutil v5.31 using pyOpenSSL 24.3.0 which has had the deprecated OpenSSL.crypto.sign removed, so fails.

Upgrading gsutil to v5.33 in the toolbox pod fixes the issue.

pip install gsutil==5.33