[Opening the issue here because I am not sure if the fix should go in to the restore task itself or how we create storage directories] ##### Problem The restore task `gitlab:backup:restore` will fail if the storage directories to be backed up are nested inside other folders due to permission issues. Sample `gitlab.rb` to reproduce and logs below 1. gitlab.rb $ cat /etc/gitlab/gitlab.rb external_url 'https://gitlab.balasankarc.in' registry_external_url 'https://registry.balasankarc.in' mattermost_external_url 'https://mattermost.balasankarc.in' git_data_dirs({ "default" => { "path" => "/var/gitlab/git-data" } }) gitlab_rails['shared_path'] = '/var/gitlab/shared' gitlab_rails['uploads_directory'] = '/var/gitlab/uploads' gitlab_ci['builds_directory'] = '/var/gitlab/builds' gitlab_rails['registry_path'] = '/var/gitlab/registry' mattermost['file_directory'] = '/var/gitlab/mattermost' 1. Contents of /var before reconfigure $ ls -l /var total 36 drwxr-xr-x 2 root root 4096 Apr 12 2016 backups drwxr-xr-x 11 root root 4096 Jan 29 06:09 cache drwxr-xr-x 23 root root 4096 Jan 29 06:09 lib drwxrwsr-x 2 root staff 4096 Apr 12 2016 local lrwxrwxrwx 1 root root 9 Dec 1 21:52 lock -> /run/lock drwxr-xr-x 8 root root 4096 Jan 29 06:09 log drwxrwsr-x 2 root mail 4096 Dec 1 21:52 mail drwxr-xr-x 2 root root 4096 Dec 1 21:52 opt lrwxrwxrwx 1 root root 4 Dec 1 21:52 run -> /run drwxr-xr-x 2 root root 4096 Dec 1 21:52 spool drwxrwxrwt 2 root root 4096 Dec 1 21:53 tmp 1. Reconfigure ran successfully 1. Contents of /var after reconfigure ls -l /var total 40 drwxr-xr-x 2 root root 4096 Apr 12 2016 backups drwxr-xr-x 11 root root 4096 Jan 29 06:09 cache drwxr-xr-x 8 root root 4096 Jan 29 06:18 gitlab drwxr-xr-x 23 root root 4096 Jan 29 06:09 lib drwxrwsr-x 2 root staff 4096 Apr 12 2016 local lrwxrwxrwx 1 root root 9 Dec 1 21:52 lock -> /run/lock drwxr-xr-x 8 root root 4096 Jan 29 06:16 log drwxrwsr-x 2 root mail 4096 Dec 1 21:52 mail drwxr-xr-x 3 root root 4096 Jan 29 06:16 opt lrwxrwxrwx 1 root root 4 Dec 1 21:52 run -> /run drwxr-xr-x 2 root root 4096 Dec 1 21:52 spool drwxrwxrwt 2 root root 4096 Dec 1 21:53 tmp $ ls -l /var/gitlab total 24 drwx------ 2 git root 4096 Jan 29 06:16 builds drwx------ 3 git root 4096 Jan 29 06:16 git-data drwxr-xr-x 2 mattermost root 4096 Jan 29 06:20 mattermost drwxr-x--- 2 registry git 4096 Jan 29 06:18 registry drwxr-x--x 5 git gitlab-www 4096 Jan 29 06:16 shared drwx------ 2 git root 4096 Jan 29 06:16 uploads 1. Created a group and a project under root and added a LICENSE file to it 1. Ran backup task $ gitlab-rake gitlab:backup:restore Unpacking backup ... done Before restoring the database, we will remove all existing tables to avoid future upgrade problems. Be aware that if you have custom tables in the GitLab database these tables and all data will be removed. Do you want to continue (yes/no)? yes Removing all tables. Press `Ctrl-C` within 5 seconds to abort Cleaning the database ... done Restoring database ... Restoring PostgreSQL database gitlabhq_production ... SET < DB related log > done Restoring repositories ... * root/test ... [DONE] Put GitLab hooks in repositories dirs [DONE] done Restoring uploads ... rake aborted! Errno::EACCES: Permission denied @ rb_file_s_rename - (/var/gitlab/uploads, /var/gitlab/uploads.1517208020) /opt/gitlab/embedded/service/gitlab-rails/lib/backup/files.rb:46:in `backup_existing_files_dir' /opt/gitlab/embedded/service/gitlab-rails/lib/backup/files.rb:37:in `restore' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:145:in `block (4 levels) in <top (required)>' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:62:in `block (3 levels) in <top (required)>' /opt/gitlab/embedded/bin/bundle:23:in `load' /opt/gitlab/embedded/bin/bundle:23:in `<main>' Tasks: TOP => gitlab:backup:uploads:restore (See full trace by running task with --trace) #### Observations 1. Permissions on `/var/gitlab` is as follows $ ls -ld /var/gitlab/ drwxr-xr-x 8 root root 4096 Jan 29 06:34 /var/gitlab/ 2. That directory is writable only to `root` 3. Restore task tries to rename the existing directories to `<name>.<timestamp>` format 4. Since rename is essentially a directory creation, it fails as `git` user can't write to `/var/gitlab` A possible solutions I see (there may be other, better ones) is to modify restore task to use a different location to temporarily move the existing directories. This can even be user interactive. That is, check if the parent directory is writable and if not, ask the user to provide a different location that they are sure is world writable I also thought about modifying how we do storage directories to set write flag on the parent directory too. But that can not be done because if the storage directory is something like `/gitlab`, it will require changing the permission on `/`, which is a no-go. /cc @gitlab-build-team