Restore task should not move existing files from Object Storage if tarball is from omnibus-gitlab package based installation
Summary
TLDR:
There is no parity between tarballs created from Helm charts based installations and those from omnibus-gitlab/source installations. The former contains files from object storage while the latter doesn't. The backup-utility in task-runner was written with the former in mind.
Restore task in Helm charts based installations overwrite existing content not only from disk, but also from Object Storage buckets. For migrating from omnibus-gitlab based installations, we are recommending users to move their files to object storage first and then create backup tarball and run the restore command in task-runner pod. However, this restore task will move the files users just migrated to a temporary bucket, thus making the whole thing a waste.
This is essentially because backup tarballs created from source and omnibus-gitlab installations are different from that made from Helm charts based installation. The former ones doesn't include files from object-storage, but only the ones present in disk. The latter, however pulls down everything from object storage and bundles it as part of the tarball.
We don't include them in omnibus-gitlab installation's tarball because object storage exists external to the GitLab instance and user's can tear down the GitLab instance without touching the object storage service. But in Helm charts, if users are using built-in Minio, if they delete the deployment, minio also gets tore down. So, it makes sense to include them in the tarball for Helm chart based installation, as long as we are providing a minio instance that users may use. (If that wasn't the case and using external object storage was mandatory, it can be dropped here also).
Scenarios
There are three scenarios when someone wants to restore a tarball to a Helm chart based deployment
- Migrating from omnibus-gitlab: The Helm chart based deployment is fresh and there is nothing to be lost while unpacking the tarball.
- Migrating from one Helm chart based deployment to a fresh one: The new Helm chart based deployment is fresh and there is nothing to be lost while unpacking the tarball.
- Restoring a tarball to an existing Helm chart based deployment because something broke and we need to go back to a known correct state: The Helm chart based deployment has data that needs to be moved before unpacking the tarball.
Proposed fix (updated)
Add installation type information to the backup_information.yml file. If it doesn't say gitlab-helm-charts, skip object storage stuff.
@Ahmadposten @marin @twk3 @WarheadsSE What do you think? We somehow needs to tell the restore task not to touch object storage if users are restoring a tarball from omnibus-gitlab package.
Proposed fix (original)
Add a --omnibus
flag to tell the backup-utility
"This is from omnibus-gitlab; don't bother with touching object storage stuff". Upon seeing this flag, we skip everything related to OS (stuff that I attempted to remove in gitlab-org/build/CNG!146 (closed))