"docker exec -it gitlab update-permissions" does not fix incorrect permissions in "/var/opt/gitlab/git-data/repositories"
Summary
After a manual intervention in the server hosting local GitLab CE Docker installation, startup of GitLab fails due to:
2017-11-22T09:13:30.296018330Z Error executing action `run` on resource 'ruby_block[directory resource: /var/opt/gitlab/git-data/repositories]'
2017-11-22T09:13:30.296166897Z ================================================================================
2017-11-22T09:13:30.296275690Z
2017-11-22T09:13:30.296843743Z Mixlib::ShellOut::ShellCommandFailed
2017-11-22T09:13:30.296876702Z ------------------------------------
2017-11-22T09:13:30.296981646Z Failed asserting that mode permissions on "/var/opt/gitlab/git-data/repositories" is 2770
And running suggested solution in log does not fix the issue:
2017-11-22T09:13:17.170648039Z If this container fails to start due to permission problems try to fix it by executing:
2017-11-22T09:13:17.170654672Z
2017-11-22T09:13:17.170660508Z docker exec -it gitlab update-permissions
2017-11-22T09:13:17.170666846Z docker restart gitlab
Steps to reproduce
- Stop GitLab
docker stop gitlab
- Change git-data permissions to something different than 2770:
docker exec -it gitlab chmod -R 0770 /var/opt/gitlab/git-data
- Start GitLab
docker start gitlab
- Same error
- Fix permissions manually just for the directory mentioned in the logs:
docker exec -it gitlab chmod -R 2770 /var/opt/gitlab/git-data/repositories
- Start GitLab
docker start gitlab
- GitLab now starts correctly
Example Project
Use default Docker installation instructions
What is the current bug behavior?
"docker exec -it gitlab update-permissions" does not fix permissions issues in "/var/opt/gitlab/git-data/repositories"
What is the expected correct behavior?
"docker exec -it gitlab update-permissions" shall fix permissions issues also in "/var/opt/gitlab/git-data/repositories"
Relevant logs and/or screenshots
2017-11-22T09:13:20.201893028Z gitlab preinstall:
2017-11-22T09:13:20.201972715Z gitlab preinstall: This node does not appear to be running a database
2017-11-22T09:13:20.201986949Z gitlab preinstall: Skipping version check, if you think this is an error exit now
2017-11-22T09:13:20.201999486Z gitlab preinstall:
2017-11-22T09:13:20.202542789Z Configuring GitLab...
2017-11-22T09:13:30.295856167Z
2017-11-22T09:13:30.295899631Z ================================================================================
2017-11-22T09:13:30.296018330Z Error executing action `run` on resource 'ruby_block[directory resource: /var/opt/gitlab/git-data/repositories]'
2017-11-22T09:13:30.296166897Z ================================================================================
2017-11-22T09:13:30.296275690Z
2017-11-22T09:13:30.296843743Z Mixlib::ShellOut::ShellCommandFailed
2017-11-22T09:13:30.296876702Z ------------------------------------
2017-11-22T09:13:30.296981646Z Failed asserting that mode permissions on "/var/opt/gitlab/git-data/repositories" is 2770
2017-11-22T09:13:30.296989817Z ---- Begin output of set -x && [ "$(stat --printf='%04a' $(readlink -f /var/opt/gitlab/git-data/repositories) | grep -o '....$')" = '2770' ] ----
2017-11-22T09:13:30.296995385Z STDOUT:
2017-11-22T09:13:30.296999380Z STDERR: + + readlink -f /var/opt/gitlab/git-data/repositories
2017-11-22T09:13:30.297098782Z grep -o ....$
2017-11-22T09:13:30.297105444Z + stat --printf=%04a /var/opt/gitlab/git-data/repositories
2017-11-22T09:13:30.297109629Z + [ 0770 = 2770 ]
2017-11-22T09:13:30.297113476Z ---- End output of set -x && [ "$(stat --printf='%04a' $(readlink -f /var/opt/gitlab/git-data/repositories) | grep -o '....$')" = '2770' ] ----
2017-11-22T09:13:30.297201968Z Ran set -x && [ "$(stat --printf='%04a' $(readlink -f /var/opt/gitlab/git-data/repositories) | grep -o '....$')" = '2770' ] returned 1
2017-11-22T09:13:30.297209773Z
2017-11-22T09:13:30.297292706Z Cookbook Trace:
2017-11-22T09:13:30.297299391Z ---------------
2017-11-22T09:13:30.297395442Z /opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/storage_directory_helper.rb:125:in `validate_command'
2017-11-22T09:13:30.297404675Z /opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/storage_directory_helper.rb:113:in `block in validate'
2017-11-22T09:13:30.297496461Z /opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/storage_directory_helper.rb:112:in `each_index'
2017-11-22T09:13:30.297521218Z /opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/storage_directory_helper.rb:112:in `validate'
2017-11-22T09:13:30.297526246Z /opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/storage_directory_helper.rb:87:in `validate!'
2017-11-22T09:13:30.297582084Z /opt/gitlab/embedded/cookbooks/cache/cookbooks/package/definitions/storage_directory.rb:35:in `block (3 levels) in from_file'
2017-11-22T09:13:30.297589499Z
2017-11-22T09:13:30.297677560Z Resource Declaration:
2017-11-22T09:13:30.297684273Z ---------------------
2017-11-22T09:13:30.297773680Z # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/package/definitions/storage_directory.rb
2017-11-22T09:13:30.297780563Z
2017-11-22T09:13:30.297871594Z 26: ruby_block "directory resource: #{params[:path]}" do
2017-11-22T09:13:30.297878657Z 27: block do
2017-11-22T09:13:30.297882518Z 28: # Ensure the directory exists
2017-11-22T09:13:30.297886480Z 29: storage_helper.ensure_directory_exists(params[:path])
2017-11-22T09:13:30.297993029Z 30:
2017-11-22T09:13:30.298392228Z 31: # Ensure the permissions are set
2017-11-22T09:13:30.298424442Z 32: storage_helper.ensure_permissions_set(params[:path])
2017-11-22T09:13:30.298429281Z 33:
2017-11-22T09:13:30.298534206Z 34: # Error out if we have not achieved the target permissions
2017-11-22T09:13:30.298541350Z 35: storage_helper.validate!(params[:path])
2017-11-22T09:13:30.298545456Z 36: end
2017-11-22T09:13:30.298549350Z 37: not_if { storage_helper.validate(params[:path]) }
2017-11-22T09:13:30.298553463Z 38: end
2017-11-22T09:13:30.298640493Z 39: end
2017-11-22T09:13:30.298646950Z
2017-11-22T09:13:30.298736348Z Compiled Resource:
2017-11-22T09:13:30.298743068Z ------------------
2017-11-22T09:13:30.298831240Z # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/package/definitions/storage_directory.rb:26:in `block in from_file'
2017-11-22T09:13:30.298838314Z
2017-11-22T09:13:30.298933842Z ruby_block("directory resource: /var/opt/gitlab/git-data/repositories") do
2017-11-22T09:13:30.298940955Z params {:path=>"/var/opt/gitlab/git-data/repositories", :owner=>"git", :group=>nil, :mode=>"2770", :name=>"/var/opt/gitlab/git-data/repositories"}
2017-11-22T09:13:30.298947305Z action [:run]
2017-11-22T09:13:30.299034624Z retries 0
2017-11-22T09:13:30.299041128Z retry_delay 2
2017-11-22T09:13:30.299045051Z default_guard_interpreter :default
2017-11-22T09:13:30.299059993Z block_name "directory resource: /var/opt/gitlab/git-data/repositories"
2017-11-22T09:13:30.299148814Z declared_type :ruby_block
2017-11-22T09:13:30.299155566Z cookbook_name "gitlab"
2017-11-22T09:13:30.299159673Z recipe_name "gitlab-shell"
2017-11-22T09:13:30.299249755Z block #<Proc:0x0000000004fc40a0@/opt/gitlab/embedded/cookbooks/cache/cookbooks/package/definitions/storage_directory.rb:27>
2017-11-22T09:13:30.299257189Z not_if { #code block }
2017-11-22T09:13:30.299261163Z end
2017-11-22T09:13:30.299848338Z
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: Current User: git Using RVM: no Ruby Version: 2.3.5p376 Gem Version: 2.6.13 Bundler Version:1.13.7 Rake Version: 12.1.0 Redis Version: 3.2.5 Git Version: 2.13.6 Sidekiq Version:5.0.4 Go Version: unknownGitLab information Version: 10.1.4 Revision: c887c03 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: postgresql URL: https://gitlab.raichuserver.binomial.es:8443 HTTP Clone URL: https://gitlab.raichuserver.binomial.es:8443/some-group/some-project.git SSH Clone URL: ssh://git@gitlab.raichuserver.binomial.es:8022/some-group/some-project.git Using LDAP: no Using Omniauth: no
GitLab Shell Version: 5.9.3 Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab Shell ...GitLab Shell version >= 5.9.3 ? ... OK (5.9.3)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:root, or git:git?
default... yes
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ...
3/1 ... repository is empty
3/2 ... ok
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Redis available via internal API: OKAccess to /var/opt/gitlab/.ssh/authorized_keys: OK
gitlab-shell self-check successfulChecking GitLab Shell ... Finished
Checking Sidekiq ...
Running? ... yes
Number of Sidekiq processes ... 1Checking Sidekiq ... Finished
Reply by email is disabled in config/gitlab.yml
Checking LDAP ...LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab ...
Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... skipped (no tmp uploads folder yet) Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
3/1 ... yes
3/2 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.3.3 ? ... yes (2.3.5)
Git version >= 2.7.3 ? ... yes (2.13.6)
Git user has default SSH configuration? ... yes
Active users: ... 2Checking GitLab ... Finished