SSH fails with fast lookup on RHEL 8 based distros
Summary
When configuring fast lookup of SSH keys per https://docs.gitlab.com/ee/administration/operations/fast_ssh_key_lookup.html#fast-lookup-is-required-for-geo-premium, once the SSH key is commented, or removed, SSH attempts to the git
user fail with Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
.
Checking the audit logs shows it is hitting SELinux is preventing gitlab-shell-au from name_connect access on the tcp_socket port 8080
Steps to reproduce
On a RHEL 8 based distro, I tested on CentOS 8:
Add:
Match User git # Apply the AuthorizedKeysCommands to the git user only
AuthorizedKeysCommand /opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-shell-authorized-keys-check git %u %k
AuthorizedKeysCommandUser git
Match all # End match, settings apply to all users again
To sshd_config
, and reload sshd.
Then comment out or delete the line in /var/opt/gitlab/.ssh/authorized_keys
pertaining to the SSH key used.
Attempt git push/pull/clone command, or try ssh git@example.com
.
This will result in an error: Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
.
Example Project
n/a
What is the current bug behavior?
SSH is failing on an SE linux policy, when SE linux is enforcing.
What is the expected correct behavior?
SSH command/git commands should work, provided the key matches.
Relevant logs and/or screenshots
Output from sealert
command:
--------------------------------------------------------------------------------
SELinux is preventing gitlab-shell-au from name_connect access on the tcp_socket port 8080.
***** Plugin catchall (100. confidence) suggests **************************
If you believe that gitlab-shell-au should be allowed name_connect access on the port 8080 tcp_socket by default.
Then you should report this as a bug.
You can generate a local policy module to allow this access.
Do
allow this access for now by executing:
# ausearch -c 'gitlab-shell-au' --raw | audit2allow -M my-gitlabshellau
# semodule -X 300 -i my-gitlabshellau.pp
Additional Information:
Source Context system_u:system_r:sshd_t:s0-s0:c0.c1023
Target Context system_u:object_r:http_cache_port_t:s0
Target Objects port 8080 [ tcp_socket ]
Source gitlab-shell-au
Source Path gitlab-shell-au
Port 8080
Host <Unknown>
Source RPM Packages
Target RPM Packages
Policy RPM selinux-policy-3.14.3-41.el8_2.5.noarch
Selinux Enabled True
Policy Type targeted
Enforcing Mode Enforcing
Host Name instance-1
Platform Linux instance-1 4.18.0-193.14.2.el8_2.x86_64 #1
SMP Sun Jul 26 03:54:29 UTC 2020 x86_64 x86_64
Alert Count 3
First Seen 2020-08-13 16:26:31 UTC
Last Seen 2020-08-13 16:35:23 UTC
Local ID 4e94513c-da7b-4d01-988a-3008103d4aef
Raw Audit Messages
type=AVC msg=audit(1597336523.847:696): avc: denied { name_connect } for pid=49166 comm="gitlab-shell-au" dest=8080 scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:http_cache_port_t:s0 tclass=tcp_socket permissive=0
Hash: gitlab-shell-au,sshd_t,http_cache_port_t,tcp_socket,name_connect
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
```bash [root@instance-1 ~]# gitlab-rake gitlab:env:info System information System: Proxy: no Current User: git Using RVM: no Ruby Version: 2.6.6p146 Gem Version: 2.7.10 Bundler Version:1.17.3 Rake Version: 12.3.3 Redis Version: 5.0.9 Git Version: 2.27.0 Sidekiq Version:5.2.9 Go Version: unknown GitLab information Version: 13.2.4-ee Revision: 5f3e8be35c3 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 11.7 URL: https://test.johnismy.name HTTP Clone URL: https://test.johnismy.name/some-group/some-project.git SSH Clone URL: git@test.johnismy.name:some-group/some-project.git Elasticsearch: no Geo: no Using LDAP: no Using Omniauth: yes Omniauth Providers: GitLab Shell Version: 13.3.0 Repository storage paths: - default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git ```
Results of GitLab application Check
Expand for output related to the GitLab application check
[root@instance-1 ~]# gitlab-rake gitlab:check SANITIZE=true Checking GitLab subtasks ... Checking GitLab Shell ... GitLab Shell: ... GitLab Shell version >= 13.3.0 ? ... OK (13.3.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful Checking GitLab Shell ... Finished Checking Gitaly ... Gitaly: ... default ... OK Checking Gitaly ... Finished Checking Sidekiq ... Sidekiq: ... Running? ... yes Number of Sidekiq processes ... 1 Checking Sidekiq ... Finished Checking Incoming Email ... Incoming Email: ... Reply by email is disabled in config/gitlab.yml Checking Incoming Email ... Finished Checking LDAP ... LDAP: ... LDAP is disabled in config/gitlab.yml Checking LDAP ... Finished Checking GitLab App ... Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... skipped (no tmp uploads folder yet) Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 2/1 ... yes Redis version >= 4.0.0? ... yes Ruby version >= 2.5.3 ? ... yes (2.6.6) Git version >= 2.22.0 ? ... yes (2.27.0) Git user has default SSH configuration? ... yes Active users: ... 2 Is authorized keys file accessible? ... yes GitLab configured to store new projects in hashed storage? ... yes All projects are in hashed storage? ... yes Elasticsearch version 6.x - 7.x? ... skipped (elasticsearch is disabled) Checking GitLab App ... Finished Checking GitLab subtasks ... Finished
Possible fixes
Add a rhel8
section to the files:
https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/master/files/gitlab-cookbooks/gitlab/recipes/selinux.rb
https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-cookbooks/package/libraries/helpers/redhat_helper.rb