SSH fails with fast lookup on RHEL 8 based distros

Summary

When configuring fast lookup of SSH keys per https://docs.gitlab.com/ee/administration/operations/fast_ssh_key_lookup.html#fast-lookup-is-required-for-geo-premium, once the SSH key is commented, or removed, SSH attempts to the git user fail with Permission denied (publickey,gssapi-keyex,gssapi-with-mic)..

Checking the audit logs shows it is hitting SELinux is preventing gitlab-shell-au from name_connect access on the tcp_socket port 8080

Steps to reproduce

On a RHEL 8 based distro, I tested on CentOS 8:

Add:

Match User git    # Apply the AuthorizedKeysCommands to the git user only
  AuthorizedKeysCommand /opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-shell-authorized-keys-check git %u %k
  AuthorizedKeysCommandUser git
Match all    # End match, settings apply to all users again

To sshd_config, and reload sshd.

Then comment out or delete the line in /var/opt/gitlab/.ssh/authorized_keys pertaining to the SSH key used.

Attempt git push/pull/clone command, or try ssh git@example.com.

This will result in an error: Permission denied (publickey,gssapi-keyex,gssapi-with-mic)..

Example Project

n/a

What is the current bug behavior?

SSH is failing on an SE linux policy, when SE linux is enforcing.

What is the expected correct behavior?

SSH command/git commands should work, provided the key matches.

Relevant logs and/or screenshots

Output from sealert command:

--------------------------------------------------------------------------------

SELinux is preventing gitlab-shell-au from name_connect access on the tcp_socket port 8080.

*****  Plugin catchall (100. confidence) suggests   **************************

If you believe that gitlab-shell-au should be allowed name_connect access on the port 8080 tcp_socket by default.
Then you should report this as a bug.
You can generate a local policy module to allow this access.
Do
allow this access for now by executing:
# ausearch -c 'gitlab-shell-au' --raw | audit2allow -M my-gitlabshellau
# semodule -X 300 -i my-gitlabshellau.pp


Additional Information:
Source Context                system_u:system_r:sshd_t:s0-s0:c0.c1023
Target Context                system_u:object_r:http_cache_port_t:s0
Target Objects                port 8080 [ tcp_socket ]
Source                        gitlab-shell-au
Source Path                   gitlab-shell-au
Port                          8080
Host                          <Unknown>
Source RPM Packages
Target RPM Packages
Policy RPM                    selinux-policy-3.14.3-41.el8_2.5.noarch
Selinux Enabled               True
Policy Type                   targeted
Enforcing Mode                Enforcing
Host Name                     instance-1
Platform                      Linux instance-1 4.18.0-193.14.2.el8_2.x86_64 #1
                              SMP Sun Jul 26 03:54:29 UTC 2020 x86_64 x86_64
Alert Count                   3
First Seen                    2020-08-13 16:26:31 UTC
Last Seen                     2020-08-13 16:35:23 UTC
Local ID                      4e94513c-da7b-4d01-988a-3008103d4aef

Raw Audit Messages
type=AVC msg=audit(1597336523.847:696): avc:  denied  { name_connect } for  pid=49166 comm="gitlab-shell-au" dest=8080 scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:http_cache_port_t:s0 tclass=tcp_socket permissive=0


Hash: gitlab-shell-au,sshd_t,http_cache_port_t,tcp_socket,name_connect

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info


```bash
[root@instance-1 ~]# gitlab-rake gitlab:env:info

System information
System:
Proxy:		no
Current User:	git
Using RVM:	no
Ruby Version:	2.6.6p146
Gem Version:	2.7.10
Bundler Version:1.17.3
Rake Version:	12.3.3
Redis Version:	5.0.9
Git Version:	2.27.0
Sidekiq Version:5.2.9
Go Version:	unknown

GitLab information
Version:	13.2.4-ee
Revision:	5f3e8be35c3
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	PostgreSQL
DB Version:	11.7
URL:		https://test.johnismy.name
HTTP Clone URL:	https://test.johnismy.name/some-group/some-project.git
SSH Clone URL:	git@test.johnismy.name:some-group/some-project.git
Elasticsearch:	no
Geo:		no
Using LDAP:	no
Using Omniauth:	yes
Omniauth Providers:

GitLab Shell
Version:	13.3.0
Repository storage paths:
- default: 	/var/opt/gitlab/git-data/repositories
GitLab Shell path:		/opt/gitlab/embedded/service/gitlab-shell
Git:		/opt/gitlab/embedded/bin/git
```

Results of GitLab application Check

Expand for output related to the GitLab application check


[root@instance-1 ~]# gitlab-rake gitlab:check SANITIZE=true
Checking GitLab subtasks ...

Checking GitLab Shell ...

GitLab Shell: ... GitLab Shell version >= 13.3.0 ? ... OK (13.3.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Gitaly ...

Gitaly: ... default ... OK

Checking Gitaly ... Finished

Checking Sidekiq ...

Sidekiq: ... Running? ... yes
Number of Sidekiq processes ... 1

Checking Sidekiq ... Finished

Checking Incoming Email ...

Incoming Email: ... Reply by email is disabled in config/gitlab.yml

Checking Incoming Email ... Finished

Checking LDAP ...

LDAP: ... LDAP is disabled in config/gitlab.yml

Checking LDAP ... Finished

Checking GitLab App ...

Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... skipped (no tmp uploads folder yet)
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
2/1 ... yes
Redis version >= 4.0.0? ... yes
Ruby version >= 2.5.3 ? ... yes (2.6.6)
Git version >= 2.22.0 ? ... yes (2.27.0)
Git user has default SSH configuration? ... yes
Active users: ... 2
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes
Elasticsearch version 6.x - 7.x? ... skipped (elasticsearch is disabled)

Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished

Possible fixes

Add a rhel8 section to the files:

https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/master/files/gitlab-cookbooks/gitlab/recipes/selinux.rb
https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-cookbooks/package/libraries/helpers/redhat_helper.rb