Draft: migrate gitlab-toolbox from gsutil to gcloud storage

What does this MR do?

Moves the GCS backend in the gitlab-toolbox image (Debian and UBI) from the legacy gsutil CLI to gcloud storage, as proposed in #2483.

• Removes gsutil==5.37 and crcmod from the pip install.
• Installs the Google Cloud CLI 590.0.1 from Google's versioned Linux tarball, with an SHA256 checksum pinned per architecture.
• Changes backup-utility and object_storage_backup.rb to gcloud storage. Only the gcs code paths change; S3 and Azure are untouched.

Changes

• gitlab-toolbox/Dockerfile (Debian):
  • Drops ARG GSUTIL_VERSION and gsutil crcmod from pip. awscli, s3cmd, pyOpenSSL and cryptography are unchanged.
  • Adds a gcloud build stage modeled on the azcopy stage. It downloads google-cloud-cli-<version>-linux-<arch>.tar.gz, verifies it with sha256sum -c, and runs install.sh --usage-reporting false.
  • The stage sets installation properties to disable update checks and survey prompts. It removes the bq component and the gsutil copy that ships inside the Cloud CLI, and strips caches and __pycache__.
  • The final image copies /usr/local/google-cloud-sdk and links gcloud into /usr/local/bin.
• gitlab-toolbox/Dockerfile.build.ubi: same pip removal and the same install, verify and trim steps in the build stage. Adds /usr/local/google-cloud-sdk and the gcloud symlink to the /assets copy.
• ci_files/variables.yml: adds GCLOUD_VERSION, GCLOUD_CHECKSUM_SHA256_AMD64 and GCLOUD_CHECKSUM_SHA256_ARM64.
• .gitlab/ci/common.gitlab-ci.yml: passes the three pins as --build-arg to the toolbox asset and final builds (EE and CE), following the CFSSL pattern.
• scripts/renovate/update-gcloud-checksum.sh (new): Renovate post-upgrade script, mirroring update-gomplate-checksum.sh. It recomputes both checksums in all three pin locations.
• gitlab-toolbox/scripts/bin/backup-utility and gitlab-toolbox/scripts/lib/object_storage_backup.rb: gcs branches only, see the mapping below. In cleanup, the empty-bucket guard now looks for listed gs:// URLs. run_cmd merges stderr into the output, so a gcloud warning would have defeated the old output.length == 0 check.
• Tests and docs:
  • development/tests/gsutil/test.sh becomes development/tests/gcloud-storage/test.sh. It authenticates with gcloud auth activate-service-account, the same command the chart runs, and keeps the wildcard ls that backup pruning depends on.
  • The gsutil test plan becomes upgrade-gcloud-storage-test-plan.md, and the test plan index is updated.

Command mapping

Caller	gsutil	gcloud storage
backup-utility	cp gs://… dst	cp gs://… dst
	cp -n	cp --no-clobber
	cp -s CLASS -n	cp --storage-class="CLASS" --no-clobber
	ls gs://B/<wildcard> (pruning)	ls gs://B/<same wildcard>, still LC_ALL=C sort
	rm URL	rm URL
object_storage_backup.rb	ls gs://B (bucket check)	ls gs://B
	-m rsync -d -x REGEX -r SRC DST	rsync --recursive --delete-unmatched-destination-objects --exclude=REGEX SRC DST
	-m rsync -r (upload, bucket to bucket)	rsync --recursive
	ls gs://B/ (cleanup)	ls gs://B/
	rm -f -r gs://B/*	rm --recursive --continue-on-error gs://B/*

-m is dropped everywhere because gcloud storage runs in parallel by default.

Authentication

No CNG change is needed. The chart's configure-gcloud-storage script runs before both the toolbox Deployment and the backup CronJob.

• When gcloud is on PATH, it runs gcloud auth activate-service-account --key-file="$GOOGLE_APPLICATION_CREDENTIALS".
• When no key is configured, it falls back to Application Default Credentials, which covers GKE Workload Identity.

Version and checksums

The checksums were computed from the downloaded tarballs; Google does not publish per-version SHA256 files.

Arch	Tarball	SHA256
amd64	google-cloud-cli-590.0.1-linux-x86_64.tar.gz	d1ae7be7e12c4e7d8286fab4774f5441020aca89dc7793ddacbace8d11d20f06
arm64	google-cloud-cli-590.0.1-linux-arm.tar.gz	fd760d8b1c2c4c484d7b659964994b07aaefddc6ac4350e681940477dbdbb22c

Image size

Compared with master, built locally for arm64 from the same base images:

Image	master	this MR	Change
Debian	3099.6 MB	3417.8 MB	+318 MB
UBI	5164.7 MB	5482.9 MB	+318 MB

The installed SDK is about 431 MB. About 101 MB of that is the Cloud CLI's bundled Python, which has to stay (see Notes for reviewers).

Test plan

Local verification:

• Debian amd64 and arm64, UBI arm64 (full image) and UBI amd64 (build stage) build successfully. In each image:
  • gcloud --version reports Google Cloud SDK 590.0.1.
  • gsutil, bq and crcmod are absent.
  • s3cmd, aws and azcopy are unchanged.
• bash -n backup-utility and ruby -c object_storage_backup.rb pass. shellcheck reports no new warnings: 104 existing warnings on both master and this branch.
• Functional run against fake-gcs-server, using the real get_existing_backups and remove_backup functions from backup-utility and the real object-storage-backup and object-storage-restore entrypoints:

Check	Result
Wildcard ls returns full gs:// URLs, sorted, and remove_backup uses them unchanged	✅
Empty bucket gives zero backups without aborting backup-utility	✅
cp --no-clobber does not overwrite; cp --storage-class sets the class; restore download works	✅
Backup rsync: --exclude drops tmp/builds; --delete-unmatched-destination-objects removes stale local files	✅
Restore rsyncs: bucket to tmp/<name>.<ts>/, and local to bucket	✅
Restore into an empty bucket: cleanup guard skips rm	✅
gcloud storage rm: objects are deleted, but the command exits 1	⚠️ Emulator issue: fake-gcs replies to DELETE with 200 null, while GCS returns 204 with an empty body

Still to do:

• Run development/tests/gcloud-storage/test.sh against a real GCS bucket, mainly to confirm gcloud storage rm exit codes.
• Back up and restore with --backend gcs on a review app or test cluster. CI's chart backups_spec.rb uses the default S3 backend (MinIO), so it does not exercise the gcs paths.
• Play the fips_pipeline bridge job (FIPS and UBI).

Notes for reviewers

• Bundled Python and FIPS: the image's Python 3.12 is built without sqlite3 (gitlab-python/Dockerfile deletes _sqlite3), and gcloud requires it. The Cloud CLI therefore keeps its bundled Python 3.14.8.
  • That Python links its own OpenSSL 3.5.9, so it does not use the system FIPS-validated module in FIPS images. This needs documenting, or sqlite3 needs to be restored in gitlab-python.
• Symlinks: gcloud storage rsync skips symlinks by default, while gsutil followed them. The data being synced always comes from object storage, which has no symlinks, so the default is kept. --no-ignore-symlinks would match gsutil exactly.
• rm --recursive also deletes noncurrent object versions. gsutil's -r did the same.
• Renovate: needs a follow-up MR in renovate-gitlab-bot (renovate/distribution/cng.config.js):
  • Replace the GoogleCloudPlatform/gsutil rule with one for GCLOUD_VERSION. A possible source is the Docker datasource gcr.io/google.com/cloudsdktool/google-cloud-cli with X.Y.Z tags.
  • Add a postUpgradeTasks rule that runs ./scripts/renovate/update-gcloud-checksum.sh, modeled on gomplate.
• Overlap with !3221 (AWS CLI v2): both MRs edit the same pip lines in the toolbox Dockerfiles, the same CI build-arg blocks, and neighboring lines in variables.yml. Whichever merges second needs a rebase.
• Unblocks hardened-builds#49 (GCS backups in the nix toolbox).

Signed-off-by: psingh29 psingh@gitlab.com

Related issues

Closes #2483

Acceptance criteria from #2483

• No gsutil call remains in gitlab-toolbox/scripts, and the images no longer install gsutil, including the copy bundled in the Cloud CLI.
• backup-utility --backend gcs backs up, restores and prunes old backups, and object-storage-backup and object-storage-restore work with GCS, on Debian and UBI. Verified against an emulator; a real GCS run is pending (see Test plan).
• The image size change is recorded: +318 MB on Debian and UBI.

Checklist

See Definition of done.

For anything in this list which will not be completed, please provide a reason in the MR discussion

Required

• Merge Request Title, and Description are up to date, accurate, and descriptive
• MR targeting the appropriate branch
• MR has a green pipeline on GitLab.com
• When ready for review, MR is labeled "~workflow::ready for review" per the Distribution MR workflow

Expected (please provide an explanation if not completing)

• Test plan indicating conditions for success has been posted and passes
• Documentation created/updated
• Integration tests added to GitLab QA
• The impact any change in container size has should be evaluated
• New dependencies are managed with GitLab forked renovatebot

🤖 Generated with Claude Code