Restoring backup corrupts uploads and artifacts

Summary

Backup restore does not restore uploads & artifacts correctly. The restore runs through without any errors, but the restored folder structure does not match the structure of the backup tar nor the original folder structure. That leads to non available avatars and missing artifacts after the restore.

Gitlab CE: 12.2.5 Install on GKE following this and that guide

Steps to reproduce

  • Take a backup using backup-utility
  • Restore the backup
  • Compare the folder structure of uploads & artifacts buckets prior and after backup 

What is the current bug behavior?

The objects from the /srv/gitlab/tmp/backups/uploads.tar.gz archive are getting restored into the uploads bucket, without creating the avatar, user, group, project sub folders within the bucket.

The same happens for the artifacts, maybe for other resources as well when they use the same mechanism .tar/gsutil

What is the expected correct behavior?

Folder structure/content is identical with the backup.tar.gz structure as well as the system the backup was taken from.

Relevant logs and/or screenshots

We have added some output to object_storage_backup.rb to see the actual commands that get executed during restore.

Restoring uploads ...
[Command] ["tar", "-xf", "/srv/gitlab/tmp/backups/uploads.tar.gz", "-C", "/srv/gitlab/tmp/backups/srv/gitlab/tmp/uploads"]
[Command] ["gsutil", "-m", "rsync", "-r", "/srv/gitlab/tmp/backups/srv/gitlab/tmp/uploads/avatar/", "gs://gitlab-uploads-test"]
[Command] ["gsutil", "-m", "rsync", "-r", "/srv/gitlab/tmp/backups/srv/gitlab/tmp/uploads/user/", "gs://gitlab-uploads-test"]
[Command] ["gsutil", "-m", "rsync", "-r", "/srv/gitlab/tmp/backups/srv/gitlab/tmp/uploads/group/", "gs://gitlab-uploads-test"]
[Command] ["gsutil", "-m", "rsync", "-r", "/srv/gitlab/tmp/backups/srv/gitlab/tmp/uploads/project/", "gs://gitlab-uploads-test"]

When we execute the commands by hand, the error is reproducible and leads to the same unstructured result. By comparing the GCS command with the s3 command we can see, that on the s3 one, a #{dir_name} gets appended to the bucket name (gs://gitlab-uploads-test => gs://gitlab-uploads-test/avatar)

When we append this on the manual execution everything works as expected and shows the correct structure.

Output of checks

(If you are reporting a bug on GitLab.com, write: This bug happens on GitLab.com)

Results of GitLab environment info

Expand for output related to the GitLab environment info 
System information
System:
Current User:   git
Using RVM:      no
Ruby Version:   2.6.3p62
Gem Version:    2.7.9
Bundler Version:1.17.3
Rake Version:   12.3.2
Redis Version:  unknown
Git Version:    unknown
Sidekiq Version:5.2.7
Go Version:     unknown


GitLab information
Version:        12.2.5
Revision:       09f8edbc
Directory:      /srv/gitlab
DB Adapter:     PostgreSQL
DB Version:     9.6.15
URL:            https://removed
HTTP Clone URL: https://removed/some-group/some-project.git
SSH Clone URL:  git@removed:some-group/some-project.git
Using LDAP:     no
Using Omniauth: yes
Omniauth Providers: google_oauth2


GitLab Shell
Version:        unknown
Repository storage paths:
  • default:      /var/opt/gitlab/repo GitLab Shell path:              /home/git/gitlab-shell Git:            /usr/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check 
Checking GitLab subtasks ...


Checking GitLab Shell ...


GitLab Shell: ... GitLab Shell version >= 9.3.0 ? ... FAIL. Please update gitlab-shell to 9.3.0 from Unknown
Running /home/git/gitlab-shell/bin/check
gitlab-shell self-check failed
  Try fixing it:
  Make sure GitLab is running;
  Check the gitlab-shell configuration file:
  sudo -u git -H editor /home/git/gitlab-shell/config.yml
  Please fix the error above and rerun the checks.


Checking GitLab Shell ... Finished


Checking Gitaly ...


Gitaly: ... default ... OK


Checking Gitaly ... Finished


Checking Sidekiq ...


Sidekiq: ... Running? ... no
  Try fixing it:
  sudo -u git -H RAILS_ENV=production bin/background_jobs start
  For more information see:
  doc/install/installation.md in section "Install Init Script"
  see log/sidekiq.log for possible errors
  Please fix the error above and rerun the checks.


Checking Sidekiq ... Finished


Checking Incoming Email ...


Incoming Email: ... Reply by email is disabled in config/gitlab.yml


Checking Incoming Email ... Finished


Checking LDAP ...


LDAP: ... LDAP is disabled in config/gitlab.yml


Checking LDAP ... Finished


Checking GitLab App ...


Git configured correctly? ... no
Trying to fix error automatically. ...Failed
  Try fixing it:
  sudo -u git -H "/usr/bin/git" config --global core.autocrlf "input"
  For more information see:
  doc/install/installation.md in section "GitLab"
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... no
  Try fixing it:
  sudo chown -R gitlab /srv/gitlab/tmp
  sudo chmod -R u+rwX /srv/gitlab/tmp
  For more information see:
  doc/install/installation.md in section "GitLab"
  Please fix the error above and rerun the checks.
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... no
  Try fixing it:
  sudo chown -R git /srv/gitlab/public/uploads
  sudo find /srv/gitlab/public/uploads -type f -exec chmod 0644 {} ;
  sudo find /srv/gitlab/public/uploads -type d -not -path /srv/gitlab/public/uploads -exec chmod 0700 {} ;
  For more information see:
  doc/install/installation.md in section "GitLab"
  Please fix the error above and rerun the checks.
Init script exists? ... no
  Try fixing it:
  Install the init script
  For more information see:
  doc/install/installation.md in section "Install Init Script"
  Please fix the error above and rerun the checks.
Init script up-to-date? ... can't check because of previous errors
Projects have namespace: ...
5/2 ... yes
3/3 ... yes
33/4 ... yes
5/5 ... yes
34/9 ... yes
21/10 ... yes
5/11 ... yes
33/12 ... yes
21/13 ... yes
21/14 ... yes
21/15 ... yes
21/16 ... yes
21/17 ... yes
51/19 ... yes
50/20 ... yes
2/23 ... yes
21/24 ... yes
52/25 ... yes
49/27 ... yes
25/30 ... yes
26/31 ... yes
53/32 ... yes
3/33 ... yes
35/34 ... yes
35/35 ... yes
47/36 ... yes
47/37 ... yes
47/38 ... yes
47/39 ... yes
25/40 ... yes
46/41 ... yes
44/43 ... yes
45/44 ... yes
21/45 ... yes
35/46 ... yes
12/47 ... yes
21/48 ... yes
48/50 ... yes
34/51 ... yes
34/52 ... yes
21/53 ... yes
3/54 ... yes
44/55 ... yes
46/56 ... yes
45/57 ... yes
21/58 ... yes
8/59 ... yes
54/60 ... yes
3/61 ... yes
47/62 ... yes
3/65 ... yes
5/66 ... yes
54/67 ... yes
35/69 ... yes
35/70 ... yes
62/72 ... yes
66/79 ... yes
35/83 ... yes
54/95 ... yes
62/100 ... yes
62/101 ... yes
35/127 ... yes
134/155 ... yes
25/175 ... yes
161/184 ... yes
35/194 ... yes
62/204 ... yes
134/205 ... yes
62/267 ... yes
62/268 ... yes
62/269 ... yes
62/270 ... yes
62/271 ... yes
134/272 ... yes
Redis version >= 2.8.0? ... no
  Try fixing it:
  Update your redis server to a version >= 2.8.0
  For more information see:
  gitlab-public-wiki/wiki/Trouble-Shooting-Guide in section sidekiq
  Please fix the error above and rerun the checks.
Ruby version >= 2.5.3 ? ... yes (2.6.3)
Git version >= 2.22.0 ? ... no
Your git bin path is "/usr/bin/git"
  Try fixing it:
  Update your git to a version >= 2.22.0 from Unknown
  Please fix the error above and rerun the checks.
Git user has default SSH configuration? ... yes
Active users: ... 18


Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished

Possible fixes

https://gitlab.com/gitlab-org/build/CNG/blob/master/gitlab-task-runner/scripts/lib/object_storage_backup.rb

Line 80 Append "/#{dir_name}/" to the gsutil command bucket name

Edited by 🤖 GitLab Bot 🤖