Skip to content

Fix backup error due to file changed warning

Evgeny Litvinov requested to merge litvinov.evgeny/gitlab:7314-backup-files-noncritical-warnings into master

What does this MR do?

The Gitlab backup procedure now uses the standard tar UNIX util as underlying file archiving tool. User can have both large file repository like artifacts and intensive CI/CD working 24/7. In such case artifacts file repo is always changed and need to be in consistent state when backup happen simultaneously.

If the tar util see the file/folder changed it exits with error code "1", see http://www.gnu.org/software/tar/manual/html_section/tar_19.html#Synopsis. So the whole Gitlab backup procedure fails. If we add the --warning=no-file-changed option to tar - the error suppressed by exit code is still 1.

We have the workaround for the problem using ENV['STRATEGY'] == 'copy' (see d6584c94). But in this solution we need at least 2+ x more space just for rsync.

Proposed solution

  • If during the backup pipeline, gzip succeeds, but tar exitstatus code is 1, then continue without error.

Screenshots/console logs

See in the MR comment !42197 (comment 435815243):

Changes are tested on my staging and prod Gitlab instances: both backup and restore procedures work.

File changing case check, start in separate console simulation:

# start file changing  in cycle
cd $GITLAB_SHARED_DIR/artifacts/
file=one
while true; do dd if=/dev/urandom of="${file}" bs=1M count=10 conv=notrunc; done

Start the backup procedure and watch the backup log:

2020-10-26 07:49:52 +0000 -- Dumping artifacts ... 
Ignoring tar exit status 1 'Some files differ': tar: ./one: file changed as we read it
2020-10-26 07:49:53 +0000 -- done

..and overall backup job exit code now is 0 as expected:

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • Label as security and @ mention @gitlab-com/gl-security/appsec
  • The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • Security reports checked/validated by a reviewer from the AppSec team

Closes #7314 (closed) Closes #14160 (closed)

Edited by Michael Kozono

Merge request reports