Restore doesn't work when certain directories are mounted on NFS
Zendesk issue: https://gitlab.zendesk.com/agent/tickets/19015
New plan
mountpoint
└── gitlab-data
├── builds
├── git-data
├── home-git
├── shared
└── uploads
Note in HA docs and backup/restore docs that if using mounted disks/NFS, this type of structure is required/supported. The specific data location cannot be mounted directly.
Original request
During the restore process, we move/rename several directories and append a timestamp. This makes sense - get a backup of existing files before restoring. However, when some of these data locations are NFS mounted, it causes lots of problems. For example, GitLab.com workers mount /var/opt/gitlab/gitlab-rails/uploads
so a restore would error our with the error below. This would happen for any file data location we restore with the Rake task.
It's somewhat surprising that we've never realized this before. Restore in an HA environment has probably never worked. What can we do to make this work in HA environments?
Error is Errno::EBUSY: Device or resource busy @ sys_fail2 - (/var/opt/gitlab/gitlab-rails/uploads, /var/opt/gitlab/gitlab-rails/uploads.1467145800)
cc/ @stanhu @jacobvosmaer-gitlab What do you think?