gitlab:backup:restore task fails for nested storage paths

[Opening the issue here because I am not sure if the fix should go in to the restore task itself or how we create storage directories]

Problem

The restore task gitlab:backup:restore will fail if the storage directories to be backed up are nested inside other folders due to permission issues. Sample gitlab.rb to reproduce and logs below

1. gitlab.rb

    $ cat /etc/gitlab/gitlab.rb 
    external_url 'https://gitlab.balasankarc.in'
    registry_external_url 'https://registry.balasankarc.in'
    mattermost_external_url 'https://mattermost.balasankarc.in'
   
    git_data_dirs({ "default" => { "path" => "/var/gitlab/git-data" } })
    gitlab_rails['shared_path'] = '/var/gitlab/shared'
    gitlab_rails['uploads_directory'] = '/var/gitlab/uploads'
    gitlab_ci['builds_directory'] = '/var/gitlab/builds'
    gitlab_rails['registry_path'] = '/var/gitlab/registry'
    mattermost['file_directory'] = '/var/gitlab/mattermost'

2. Contents of /var before reconfigure

    $ ls -l /var
    total 36
    drwxr-xr-x  2 root root  4096 Apr 12  2016 backups
    drwxr-xr-x 11 root root  4096 Jan 29 06:09 cache
    drwxr-xr-x 23 root root  4096 Jan 29 06:09 lib
    drwxrwsr-x  2 root staff 4096 Apr 12  2016 local
    lrwxrwxrwx  1 root root     9 Dec  1 21:52 lock -> /run/lock
    drwxr-xr-x  8 root root  4096 Jan 29 06:09 log
    drwxrwsr-x  2 root mail  4096 Dec  1 21:52 mail
    drwxr-xr-x  2 root root  4096 Dec  1 21:52 opt
    lrwxrwxrwx  1 root root     4 Dec  1 21:52 run -> /run
    drwxr-xr-x  2 root root  4096 Dec  1 21:52 spool
    drwxrwxrwt  2 root root  4096 Dec  1 21:53 tmp

3. Reconfigure ran successfully

4. Contents of /var after reconfigure

    ls -l /var
    total 40
    drwxr-xr-x  2 root root  4096 Apr 12  2016 backups
    drwxr-xr-x 11 root root  4096 Jan 29 06:09 cache
    drwxr-xr-x  8 root root  4096 Jan 29 06:18 gitlab
    drwxr-xr-x 23 root root  4096 Jan 29 06:09 lib
    drwxrwsr-x  2 root staff 4096 Apr 12  2016 local
    lrwxrwxrwx  1 root root     9 Dec  1 21:52 lock -> /run/lock
    drwxr-xr-x  8 root root  4096 Jan 29 06:16 log
    drwxrwsr-x  2 root mail  4096 Dec  1 21:52 mail
    drwxr-xr-x  3 root root  4096 Jan 29 06:16 opt
    lrwxrwxrwx  1 root root     4 Dec  1 21:52 run -> /run
    drwxr-xr-x  2 root root  4096 Dec  1 21:52 spool
    drwxrwxrwt  2 root root  4096 Dec  1 21:53 tmp
   
    $ ls -l /var/gitlab
    total 24
    drwx------ 2 git        root       4096 Jan 29 06:16 builds
    drwx------ 3 git        root       4096 Jan 29 06:16 git-data
    drwxr-xr-x 2 mattermost root       4096 Jan 29 06:20 mattermost
    drwxr-x--- 2 registry   git        4096 Jan 29 06:18 registry
    drwxr-x--x 5 git        gitlab-www 4096 Jan 29 06:16 shared
    drwx------ 2 git        root       4096 Jan 29 06:16 uploads

5. Created a group and a project under root and added a LICENSE file to it

6. Ran backup task

    $ gitlab-rake gitlab:backup:restore
    Unpacking backup ... done
    Before restoring the database, we will remove all existing
    tables to avoid future upgrade problems. Be aware that if you have
    custom tables in the GitLab database these tables and all data will be
    removed.
   
    Do you want to continue (yes/no)? yes
    Removing all tables. Press `Ctrl-C` within 5 seconds to abort
    Cleaning the database ... 
    done
    Restoring database ... 
    Restoring PostgreSQL database gitlabhq_production ... SET
    < DB related log >
    done
    Restoring repositories ...
    * root/test ... [DONE]
    Put GitLab hooks in repositories dirs [DONE]
    done
    Restoring uploads ... 
    rake aborted!
    Errno::EACCES: Permission denied @ rb_file_s_rename - (/var/gitlab/uploads, /var/gitlab/uploads.1517208020)
    /opt/gitlab/embedded/service/gitlab-rails/lib/backup/files.rb:46:in `backup_existing_files_dir'
    /opt/gitlab/embedded/service/gitlab-rails/lib/backup/files.rb:37:in `restore'
    /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:145:in `block (4 levels) in <top (required)>'
    /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:62:in `block (3 levels) in <top (required)>'
    /opt/gitlab/embedded/bin/bundle:23:in `load'
    /opt/gitlab/embedded/bin/bundle:23:in `<main>'
    Tasks: TOP => gitlab:backup:uploads:restore
    (See full trace by running task with --trace)

Observations

1. Permissions on /var/gitlab is as follows

    $ ls -ld /var/gitlab/
    drwxr-xr-x 8 root root 4096 Jan 29 06:34 /var/gitlab/

2. That directory is writable only to root

3. Restore task tries to rename the existing directories to <name>.<timestamp> format

4. Since rename is essentially a directory creation, it fails as git user can't write to /var/gitlab

A possible solutions I see (there may be other, better ones) is to modify restore task to use a different location to temporarily move the existing directories. This can even be user interactive. That is, check if the parent directory is writable and if not, ask the user to provide a different location that they are sure is world writable

I also thought about modifying how we do storage directories to set write flag on the parent directory too. But that can not be done because if the storage directory is something like /gitlab, it will require changing the permission on /, which is a no-go.

/cc @gitlab-build-team