Bugs and Documentation Errors for NFS Implementation
NFS documented parameter user['home'] does not appear to redirect any user profiles to a shared NFS location to find authorized_keys (assuming that is part of this parameter's function) and there also appears to be implementation specific hardcoded data in the sshd_conf used by sshd.
The hardcode is actual helpful while this bug exists.
We were able to fully implemented the NFS implementation pattern using the documentation - except the key reads of authorized_keys would not be done against the NFS shared data location. We found what seems to be a hardcoded location for the authorized_keys file which helped us work around the problem.
If the below is working as designed, then the documentation needs improvement so that the NFS implementation pattern can be more easily followed.
Steps to reproduce
- Setup NFS as per the documentation here: https://docs.gitlab.com/ce/administration/high_availability/nfs.html#change-default-file-locations
- When individual users add SSH keys to their profile, SSH keys in the shared location are not updated. Work around: update gitlab_shell['auth_file'] is pointed to the shared location.
- After work around, SSH keys in the shared location are updated when added via Gitlab GUI, but they are not correctly read by gitlab when users attempt to interact using SSH keys, resulting in authorization errors.
What is the current bug behavior?
We are guessing that the user['home'] setting in the NFS documentation (referenced above) is supposed to cause the user 'git' to have it's keys located on the NFS shared location. If this is the intended way of sharing the SSH keys - it is not working to redirect SSH key file operations.
Secondarily the sshd_conf file configured by gitlab contains a hard coded secondary key file reference to the documentation's sample shared location mount point /gitlab-data, specifically /gitlab-data/ssh/authorized_keys.
What is the expected correct behavior?
- user['home'], or another parameter, should cause the shared authorized_keys file to be used by git for reads and writes on the shared NFS location.
- Any configuration files changes in sshd_conf that help the NFS shared location to work properly should either (a) be softcoded to take a variable or (b) the documentation should be updated to reflect that for NFS implementations the shared mount point must be exactly "/gitlab-data"
- Depending on how authorized_keys writes happen, the NFS documentation may need to be updated to include the parameter gitlab_shell['auth_file'] - if it is required to get writes to happen to the shared location when user['home'] is working correctly.
Relevant logs and/or screenshots
Find the user and sshd_config file in use:
[email protected]:/var/opt/gitlab# ps -ef | grep "sshd" root 1022 1015 0 Sep29 ? 00:00:00 runsv sshd root 1023 1022 0 Sep29 ? 00:00:00 svlogd -tt /var/log/gitlab/sshd root 1024 1022 0 Sep29 ? 00:00:04 /usr/sbin/sshd -D -f /assets/sshd_config -e
sshd is using the config file /assets/sshd_config, list it's contents:
[email protected]:/var/opt/gitlab# cat /assets/sshd_config Port 22 ChallengeResponseAuthentication no HostKey /etc/gitlab/ssh_host_rsa_key HostKey /etc/gitlab/ssh_host_ecdsa_key HostKey /etc/gitlab/ssh_host_ed25519_key Protocol 2 PermitRootLogin no PasswordAuthentication no MaxStartups 100:30:200 AllowUsers git PrintMotd no PrintLastLog no PubkeyAuthentication yes AuthorizedKeysFile %h/.ssh/authorized_keys /gitlab-data/ssh/authorized_keys
The sshd_config file has a hardcoded second authorized keys file as /gitoab-data/ssh/authorized_keys. We believe it is hard coded because while it was not working we did not have that location string coded into any gitlab.rb parameter nor do we manipulate sshd_config directly and yet the value appeared exactly as above.
When we forced our configuration to match this value (mount point named exactly /gitlab-data and key file location as /ssh/authorized_keys), then reads and writes to the file work fine.
Output of checks
(If you are reporting a bug on GitLab.com, write: This bug happens on GitLab.com)
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: Current User: git Using RVM: no Ruby Version: 2.3.5p376 Gem Version: 2.6.13 Bundler Version:1.13.7 Rake Version: 12.0.0 Redis Version: 3.2.5 Git Version: 2.13.5 Sidekiq Version:5.0.4 Go Version: unknown
GitLab information Version: 10.0.1 Revision: 2417795 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: postgresql URL: omitted HTTP Clone URL: omitted SSH Clone URL: [email protected]:some-group/some-project.git Using LDAP: no Using Omniauth: yes Omniauth Providers: saml
GitLab Shell Version: 5.9.0 Repository storage paths:
- default: /gitlab-data/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to GitLab application Check
[email protected]:/# gitlab-rake gitlab:check SANITIZE=true Checking GitLab Shell ...
GitLab Shell version >= 5.9.0 ? ... OK (5.9.0) Repo base directory exists? default... yes Repo storage directories are symlinks? default... no Repo paths owned by git:root, or git:git? default... yes Repo paths access is drwxrws---? default... yes hooks directories in repos are links: ... 8/4 ... ok 12/5 ... ok 12/6 ... ok 12/8 ... repository is empty 12/9 ... repository is empty 2/10 ... ok 3/11 ... ok 15/12 ... ok Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Check GitLab API access: OK Redis available via internal API: OK
Access to /gitlab-data/ssh/authorized_keys: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Sidekiq ...
Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Reply by email is disabled in config/gitlab.yml Checking LDAP ...
LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 8/4 ... yes 12/5 ... yes 12/6 ... yes 12/8 ... yes 12/9 ... yes 2/10 ... yes 3/11 ... yes 15/12 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.3.3 ? ... Exception: undefined method
run_command' for SystemCheck::App::RubyVersionCheck:Class Did you mean? run_commands Git version >= 2.7.3 ? ... Exception: undefined methodrun_command' for SystemCheck::App::GitVersionCheck:Class Did you mean? run_commands Git user has default SSH configuration? ... yes Active users: ... 7
Checking GitLab ... Finished
Before the below work around we successfully redirected gitlabs SSH key writes to the NFS location by configuring gitlab_shell['auth_file'], but until the below work around was put in place, gitlab was not reading the keys from this location.
Our work around was to match the hard coded authorized key path exactly with our implementation. That path is /gitlab-data/ssh/authorizedkeys so we made sure the NFS mount is /gitlab-data and we set gitlab_shell['auth_file'] to /gitlab-data/ssh/authorizedkeys