backup / restore in gitlab-12.0.1-ee is intermittently failing to create valid single tar backup that can be restored

Summary

After upgrading to gitlab-12, the omnibus gitlab-ee 12.0.0-ee, and 12.0.1-ee are not creating proper tar backups that can be restored into a new vm of same exact release version. Note that in our instance from Gitlab-11, we moved to hashed storage, and upgraded from Postgres 9 to 10 as we upgraded to Gitlab 12.

Steps to reproduce

  1. Take a working self-hosted omnibus version of gitlab 11.x-ee, run backups. Restore backups to 2nd VM with same version of gitlab-ee. This works.

  2. Upgrade to gitlab-12-0.1. Run same scripts to backup. Intermittently, the gitlab-rails backup command:

sudo gitlab-rake gitlab:backup:create

will create a smaller tar file than expected, and dump sub tar files (pages, artifacts, registry, db, repositories) into the backup directory, rather than a single tar file.

Example Project

Here is an example of the cron script that runs to create the backup files:

root@gitlab:~/cron# cat gitlab-backup.sh #!/bin/bash

this script will run nightly on gitlab.test.example.com, to backup, encrypt, and rsync to gitlab-san-1

set +x

command below ensures gitlab is in a good state

gitlab-ctl reconfigure

command below creates the gitlab backup on the local gitlab vm in /var/opt/gitlab/backups

gitlab-rake gitlab:backup:create

change to local backup dir ; assign the file with the latest date to fn

cd /var/opt/gitlab/backups || exit fn=$(ls -t | head -n1)

command below encrypts the latest backup and appends .enc to filename

cat $fn | openssl enc -<crypto_alg.> -pass file:/path/to/akeyfile201906a > $fn.enc

commands below rsync the local /backups directory to the gitlab-san-1

rsync -av /var/opt/gitlab/backups/ account@gitlab-san-1.example.net:/path/to/backup/gitlab/on/san logger "rsync of gitlab server directory to gitlab-san-1 completed"

What is the current bug behavior?

tar backups in gitlab 12.01 are not stable.

What is the expected correct behavior?

proper tar archive is created from gitlab rails command

Edited by 🤖 GitLab Bot 🤖