Backups fail occasionally with "tar: ./objects: file changed as we read it" error

Summary

Backups fail occasionally with "tar: ./objects: file changed as we read it" error which is probably caused by taking the backup with tar instead of using git bundle.

Steps to reproduce

  1. take a backup: gitlab-rake gitlab:backup:create
  2. perform some operation on the git repository
  3. observe the backup failing

(This is probably a race condition due to using tar instead of git bundle so this does not reproduce all the time.)

Expected behavior

Backups not to fail

Relevant logs and/or screenshots

Backup failed
[FAILED]
failed: tar -cf /var/opt/gitlab/backups/repositories/path/to.bundle -C /var/opt/gitlab/git-data/repositories/path/to.git .
tar: ./objects: file changed as we read it

Output of checks

Results of GitLab Application Check

All tests passed.

Results of GitLab Environment Info

omnibus-gitlab CE

System information
System:         Debian 7.9
Current User:   git
Using RVM:      no
Ruby Version:   2.1.7p400
Gem Version:    2.2.5
Bundler Version:1.10.6
Rake Version:   10.4.2
Sidekiq Version:3.3.0

GitLab information
Version:        8.2.3
Revision:       19daba8
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     postgresql
URL:            https://gitlab.XX
HTTP Clone URL: https://gitlab.XXX.com/some-group/some-project.git
SSH Clone URL:  git@gitlab.XXX:some-group/some-project.git
Using LDAP:     yes
Using Omniauth: no

GitLab Shell
Version:        2.6.8
Repositories:   /var/opt/gitlab/git-data/repositories
Hooks:          /opt/gitlab/embedded/service/gitlab

Possible fixes

I think this is a regression due to the change where backing up was changed to use tar instead of git bundle.

This was done in https://gitlab.com/gitlab-org/gitlab-ce/commit/d411a9e4d8063fdc9b6d0f74cad7345245a1fb0b so that git-annex files would be properly backed up.

Discussion on why backing up a git repository with tar is not necessarily a good idea: http://marc.info/?l=git&m=136422341014631&w=2

Maybe this could be fixed by locking the repository (i.e., preventing commits/other operations) when taking backups?