Configuring Gitlab for HA with AWS EFS?
Hello,
I am following the instructions on this page for the Active/Active setup, but the difference is that I'm using AWS EFS (which is basically NFSv4.1) and I'm getting the following error when I do sudo gitlab-ctl reconfigure
Recipe: gitlab::gitlab-shell
* ruby_block[directory resource: /var/opt/gitlab/git-data] action run (skipped due to not_if)
* ruby_block[directory resource: /var/opt/gitlab/git-data/repositories] action run (skipped due to not_if)
* ruby_block[directory resource: /var/opt/gitlab/.ssh] action run (skipped due to not_if)
* directory[/opt/gitlab/embedded/service/gitlab-shell/hooks/] action create (up to date)
* directory[/var/log/gitlab/gitlab-shell/] action create (up to date)
* directory[/var/opt/gitlab/gitlab-shell] action create (up to date)
* template[/var/opt/gitlab/gitlab-shell/config.yml] action create (up to date)
* link[Link /opt/gitlab/embedded/service/gitlab-shell/config.yml to /var/opt/gitlab/gitlab-shell/config.yml] action create (up to date)
* link[/opt/gitlab/embedded/service/gitlab-shell/.gitlab_shell_secret] action create (up to date)
* execute[/opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-keys check-permissions] action run
[execute] /opt/gitlab/embedded/lib/ruby/2.3.0/logger.rb:703:in `initialize': Permission denied @ rb_sysopen - /var/log/gitlab/gitlab-shell/gitlab-shell.log (Errno::EACCES)
from /opt/gitlab/embedded/lib/ruby/2.3.0/logger.rb:703:in `open'
from /opt/gitlab/embedded/lib/ruby/2.3.0/logger.rb:703:in `open_logfile'
from /opt/gitlab/embedded/lib/ruby/2.3.0/logger.rb:695:in `set_dev'
from /opt/gitlab/embedded/lib/ruby/2.3.0/logger.rb:635:in `initialize'
from /opt/gitlab/embedded/lib/ruby/2.3.0/logger.rb:353:in `new'
from /opt/gitlab/embedded/lib/ruby/2.3.0/logger.rb:353:in `initialize'
from /opt/gitlab/embedded/service/gitlab-shell/lib/gitlab_logger.rb:15:in `new'
from /opt/gitlab/embedded/service/gitlab-shell/lib/gitlab_logger.rb:15:in `<top (required)>'
from /opt/gitlab/embedded/service/gitlab-shell/lib/gitlab_keys.rb:4:in `require_relative'
from /opt/gitlab/embedded/service/gitlab-shell/lib/gitlab_keys.rb:4:in `<top (required)>'
from /opt/gitlab/embedded/lib/ruby/site_ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
from /opt/gitlab/embedded/lib/ruby/site_ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
from /opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-keys:24:in `<main>'
================================================================================
Error executing action `run` on resource 'execute[/opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-keys check-permissions]'
================================================================================
Mixlib::ShellOut::ShellCommandFailed
------------------------------------
Expected process to exit with [0], but received '1'
---- Begin output of /opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-keys check-permissions ----
STDOUT:
STDERR: /opt/gitlab/embedded/lib/ruby/2.3.0/logger.rb:703:in `initialize': Permission denied @ rb_sysopen - /var/log/gitlab/gitlab-shell/gitlab-shell.log (Errno::EACCES)
from /opt/gitlab/embedded/lib/ruby/2.3.0/logger.rb:703:in `open'
from /opt/gitlab/embedded/lib/ruby/2.3.0/logger.rb:703:in `open_logfile'
from /opt/gitlab/embedded/lib/ruby/2.3.0/logger.rb:695:in `set_dev'
from /opt/gitlab/embedded/lib/ruby/2.3.0/logger.rb:635:in `initialize'
from /opt/gitlab/embedded/lib/ruby/2.3.0/logger.rb:353:in `new'
from /opt/gitlab/embedded/lib/ruby/2.3.0/logger.rb:353:in `initialize'
from /opt/gitlab/embedded/service/gitlab-shell/lib/gitlab_logger.rb:15:in `new'
from /opt/gitlab/embedded/service/gitlab-shell/lib/gitlab_logger.rb:15:in `<top (required)>'
from /opt/gitlab/embedded/service/gitlab-shell/lib/gitlab_keys.rb:4:in `require_relative'
from /opt/gitlab/embedded/service/gitlab-shell/lib/gitlab_keys.rb:4:in `<top (required)>'
from /opt/gitlab/embedded/lib/ruby/site_ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
from /opt/gitlab/embedded/lib/ruby/site_ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
from /opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-keys:24:in `<main>'
---- End output of /opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-keys check-permissions ----
Ran /opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-keys check-permissions returned 1
Resource Declaration:
---------------------
# In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/gitlab-shell.rb
115: execute "#{gitlab_shell_keys_check} check-permissions" do
116: user git_user
117: group git_group
118: end
119:
Compiled Resource:
------------------
# Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/gitlab-shell.rb:115:in `from_file'
execute("/opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-keys check-permissions") do
action [:run]
retries 0
retry_delay 2
default_guard_interpreter :execute
command "/opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-keys check-permissions"
backup 5
group "git"
returns 0
user "git"
declared_type :execute
cookbook_name "gitlab"
recipe_name "gitlab-shell"
end
Platform:
---------
x86_64-linuxI've tried to do the following:
/etc/fstab:
10.96.xx.xxx:/gitlab/.ssh /var/opt/gitlab/.ssh nfs defaults,vers=4.1,soft,rsize=1048576,wsize=1048576,lookupcache=positive 0 2
10.96.xx.xxx:/gitlab/gitlab-rails/uploads /var/opt/gitlab/gitlab-rails/uploads nfs defaults,vers=4.1,soft,rsize=1048576,wsize=1048576,lookupcache=positive 0 2
10.96.xx.xxx:/gitlab/gitlab-rails/shared /var/opt/gitlab/gitlab-rails/shared nfs defaults,vers=4.1,soft,rsize=1048576,wsize=1048576,lookupcache=positive 0 2
10.96.xx.xxx:/gitlab/gitlab-ci/builds /var/opt/gitlab/gitlab-ci/builds nfs defaults,vers=4.1,soft,rsize=1048576,wsize=1048576,lookupcache=positive 0 2
10.96.xx.xxx:/gitlab/git-data /var/opt/gitlab/git-data nfs defaults,vers=4.1,soft,rsize=1048576,wsize=1048576,lookupcache=positive 0 2where /var/opt/gitlab looks like this:
drwxr-xr-x 7 root root 4096 Oct 11 13:47 .
drwxr-xr-x 3 root root 4096 Oct 10 13:16 ..
-rw-r--r-- 1 git git 301 Oct 10 17:09 .gitconfig
drwx------ 3 git root 4096 Oct 10 00:53 git-data
drwxr-xr-x 3 root root 4096 Oct 10 16:59 gitlab-ci
drwxr-xr-x 4 root root 4096 Oct 10 16:59 gitlab-rails
drwx------ 2 git root 4096 Oct 11 13:47 gitlab-shell
drwx------ 2 git git 4096 Jul 22 2015 .sshand /etc/fstab:
10.96.xx.xxx:/gitlab /var/opt/gitlab nfs defaults,vers=4.1,lookupcache=positive 0 0where /var/opt/gitlab looks like:
drwxr-xr-x 17 root root 4096 Oct 10 16:55 .
drwxr-xr-x 3 root root 4096 Oct 11 14:00 ..
drwx------ 3 git root 4096 Aug 27 15:12 backups
-rw------- 1 root root 38 Jul 22 2015 bootstrapped
-rw-r--r-- 1 git git 301 Oct 10 16:55 .gitconfig
drwx------ 3 git root 4096 Oct 10 00:53 git-data
drwxr-xr-x 3 git root 4096 Sep 28 2015 gitlab-ci
drwxr-x--- 2 git git 4096 Nov 30 2015 gitlab-git-http-server
drwxr-xr-x 9 git root 4096 Sep 29 19:46 gitlab-rails
drwx------ 2 git root 4096 Oct 10 16:55 gitlab-shell
drwxr-x--- 2 git gitlab-www 4096 Sep 29 19:46 gitlab-workhorse
drwx------ 3 root root 4096 Sep 29 19:46 logrotate
drwx------ 2 root root 4096 Jul 22 2015 lost+found
drwxr-xr-x 3 996 root 4096 Sep 29 19:46 mattermost
drwxr-x--- 9 root gitlab-www 4096 Sep 26 13:13 nginx
drwx------ 3 996 root 4096 Aug 19 2015 postgresql
drwxr-x--- 2 997 git 4096 Oct 9 04:05 redis
drwx------ 2 root root 4096 Sep 29 19:46 remote-syslog
drwx------ 2 git git 4096 Jul 22 2015 .ssh
-rw-r--r-- 1 root root 40 Jul 24 05:00 trusted-certs-directory-hashI can manipulate any dir/file on the EFS when I'm root or any "owner" of the directory that I'm manipulating.
Any idea why I'm getting a permissions error?