Backup to NFS share won't release registry.tar.gz

changed the description

Setting devopsenablement based on groupdistribution.

@Vashiru can you see if the problem goes away if you switch your backup command to: gitlab-rake gitlab:backup:create ?

The gitlab-backup create command is somewhat new and might be related to the issue.

added awaiting feedback label

mentioned in issue gitlab-org/distribution/team-tasks#533 (closed)

mentioned in issue #37890 (closed)

mentioned in issue #39030 (closed)

mentioned in issue #103432 (closed)

@twk3 I did some more testing these last few days. I figured out that this:

gitlab_rails['backup_keep_time'] = 604800
gitlab_rails['backup_upload_connection'] = {
  :provider => 'Local',
  :local_root => '/mnt/nas/gitlab-server/backups'
}
gitlab_rails['backup_upload_remote_directory'] = '.'

Does not actually back-up to /mnt/nas/gitlab-server/backups, it will instead default to /var/opt/gitlab/backups. (Suppose that's an issue on it's own).

So I disabled the backup_upload_connection bit, set gitlab_rails['backup_path'] = "/mnt/nas/gitlab-server/backups/new" and tried running sudo gitlab-backup create. That wrote to registry.tar.gz for about an hour and seemingly did nothing anymore. After 24 hours I went ahead and interrupted it with Ctrl + C. Screenshot from that moment:

So it appears that gitlab-backup create CRON=1 never actually finishes, which would explain the behavior I'm seeing from the NFS share.

As a temporary solution I'll try /usr/bin/gitlab-rake gitlab:backup:create with gitlab_rails['backup_path'] = "/mnt/nas/gitlab-server/backups/new" and see how that goes.

@Vashiru did you have any better success with the rake command?

mentioned in issue #121511 (closed)

mentioned in issue #193057 (closed)

mentioned in issue #195130 (closed)

mentioned in issue #196495 (closed)

mentioned in issue #197831 (closed)

mentioned in issue #199014 (closed)

@twk3 Nope I did not. Writing to a new folder doesn't yield any better results. But I am able to confirm that the process never finishes. If I do ps aux | grep back after a few days, the output is this:

Also memory usage seems to keep rising. So there appears there's something that's keeping the registry back-up from finishes. Are there perhaps any log files I could check to figure out what's causing this?

@Vashiru I wonder how large your registry data is? Another thing to consider is to switch the backup strategy to COPY to see if it ever completes with that method: https://docs.gitlab.com/ee/raketasks/backup_restore.html#backup-strategy-option

By default registry data is stored in: /var/opt/gitlab/gitlab-rails/shared/registry

In my gitlab.rb: gitlab_rails['registry_path'] = "/mnt/nas/gitlab-server/data-docker"

Looking there I'm not sure if I should quote the data-docker directory or just the registry inside it so:

$ du -sh data-docker
359G	data-docker

$ du -sh registry
44G	registry

The 44G size seems to match with the size of the registry.tar.gz, so I assume that's the directory we're talking about and I would assume it's getting close to finishing it's back-up but not quite.

As for the copy method, I'll give that a try.

The copy strategy didn't complete either but instead crashed with a stack trace saying permission denied trying to access certain folders. I've corrected them and restarted the backup, see if that one finishes. If it does I'll also try the original method again

With the back-up strategy set to copy and the folder permissions fixed, it managed to finish the export. It became a tar file of 368 GB.

I'll switch the back-up method back to what it was (aka the default instead of copy and have that run).

mentioned in issue #201683 (closed)

Switched the back-up method back to the default from using COPY. It's now finishing correctly.

The issue boiled down to a few folders / files in the registry having the wrong owner. Running sudo chown -R registry:git on the registry folder fixed it for me.

I assume this issue has been caused by a bug in our QNAP NAS, which had flaky NFS permissions for nearly a year, rather than something inside Gitlab. Since it's resolved now, I'm closing the issue.

Thanks for your help @twk3 !

closed

Backup to NFS share won't release registry.tar.gz

Summary

Steps to reproduce

What is the current bug behavior?

What is the expected correct behavior?

Relevant logs and/or screenshots

Results of GitLab environment info

Results of GitLab application Check

Designs

Child items ...

Activity