backup / restore in gitlab-12.0.1-ee is intermittently failing to create valid single tar backup that can be restored
Summary
After upgrading to gitlab-12, the omnibus gitlab-ee 12.0.0-ee, and 12.0.1-ee are not creating proper tar backups that can be restored into a new vm of same exact release version. Note that in our instance from Gitlab-11, we moved to hashed storage, and upgraded from Postgres 9 to 10 as we upgraded to Gitlab 12.
Steps to reproduce
-
Take a working self-hosted omnibus version of gitlab 11.x-ee, run backups. Restore backups to 2nd VM with same version of gitlab-ee. This works.
-
Upgrade to gitlab-12-0.1. Run same scripts to backup. Intermittently, the gitlab-rails backup command:
sudo gitlab-rake gitlab:backup:create
will create a smaller tar file than expected, and dump sub tar files (pages, artifacts, registry, db, repositories) into the backup directory, rather than a single tar file.
Example Project
Here is an example of the cron script that runs to create the backup files:
root@gitlab:~/cron# cat gitlab-backup.sh
#!/bin/bash
this script will run nightly on gitlab.test.example.com, to backup, encrypt, and rsync to gitlab-san-1
set +x
command below ensures gitlab is in a good state
gitlab-ctl reconfigure
command below creates the gitlab backup on the local gitlab vm in /var/opt/gitlab/backups
gitlab-rake gitlab:backup:create
change to local backup dir ; assign the file with the latest date to fn
cd /var/opt/gitlab/backups || exit fn=$(ls -t | head -n1)
command below encrypts the latest backup and appends .enc to filename
cat $fn | openssl enc -<crypto_alg.> -pass file:/path/to/akeyfile201906a > $fn.enc
commands below rsync the local /backups directory to the gitlab-san-1
rsync -av /var/opt/gitlab/backups/ account@gitlab-san-1.example.net:/path/to/backup/gitlab/on/san logger "rsync of gitlab server directory to gitlab-san-1 completed"
What is the current bug behavior?
tar backups in gitlab 12.01 are not stable.
What is the expected correct behavior?
proper tar archive is created from gitlab rails command