Toolbox: backup-utility --maximum-backups no longer works with azure (since azcopy >= 10.25.0)

Summary

backup-utility always finds "0 existing backups" using the azure backend no matter how many are already in the bucket.

Steps to reproduce

  • Install a fresh gitlab helm chart configured to use the azure backend for backups
  • Run backup-utility in the toolbox container twice
  • Verify at least two backup .tar files exist via the Azure Portal GUI
  • Run backup-utility --cleanup --backend azure --maximum-backups 1 in the toolbox

Configuration used

gitlab:
  toolbox:
    backups:
      objectStorage:
        backend: azure
        config:
          key: config
          secret: gitlab-backup-azure-creds
      cron:
        enabled: true
        schedule: "0/5 * * * *"
        extraArgs: "--maximum-backups 1"

Current behavior

git@gitlab-toolbox-696b478fc-6c9f8:/$ backup-utility --cleanup --backend azure --maximum-backups 1
Found 0 existing backups. Maximum allowed is 1
[DONE] Finished pruning old backups
git@gitlab-toolbox-696b478fc-6c9f8:/$

Expected behavior

It should find at least 2 backups and prune them to leave a single backup

Versions

  • Chart: 8.1.2
  • Platform:
    • Self-hosted: microk8s
  • Kubernetes: (kubectl version)
    • Client: v1.28.10
    • Server: v1.28.9
  • Helm: (helm version)
    • Client: version.BuildInfo{Version:"v3.13.2+35.el9", GitCommit:"fa6e939d7984e1be0d6fbc2dc920b6bbcf395932", GitTreeState:"clean", GoVersion:"go1.20.12"}
    • Server:

Relevant logs

Troubleshooting

It seems that azcopy changed the default output for azcopy list in version 10.25.0 (which actually had a bug producing no output at all but now that's fixed in 10.25.1, the output that backup-utility expects is now in a different format)

It used to look like this:

INFO: 1721055606_2024_07_15_17.1.2-ee_gitlab_backup.tar;  Content Length: 1.44 MiB
INFO: 1721133907_2024_07_16_17.1.2-ee_gitlab_backup.tar;  Content Length: 2.31 MiB
INFO: 1721151014_2024_07_16_17.1.2-ee_gitlab_backup.tar;  Content Length: 3.25 MiB

but now (azcopy >= 10.25.1) looks like this:

1721055606_2024_07_15_17.1.2-ee_gitlab_backup.tar; Content Length: 1.44 MiB
1721133907_2024_07_16_17.1.2-ee_gitlab_backup.tar; Content Length: 2.31 MiB
1721151014_2024_07_16_17.1.2-ee_gitlab_backup.tar; Content Length: 3.25 MiB

Consequently https://gitlab.com/gitlab-org/build/CNG/-/blob/master/gitlab-toolbox/scripts/bin/backup-utility?ref_type=heads#L165 finds no backups.

A quick fix might be changing the awk to awk '/^[[:alnum:]_\.\-]+;/ {gsub(/;$/, "", $1); print $1}' but I note that the CHANGELOG announces proper support for --output-type=json which may be a better route to fix this permanently.

Edited by Keith White