Adding Kubernetes Cluster Integration via non-local IP fails SSL Verify
Summary
When adding a Kubernetes Cluster on our Self-Hosted Instance or even a test project on gitlab.com via a non-local IP Adress it fails
Steps to reproduce
Add a Cluster without a FQDN but just non-local IP
Example Project
Example here: https://gitlab.com/adamvb/testdirs/-/clusters/112844
What is the current bug behavior?
When adding the Cluster via a local Subnet IP e.g. https://192.168.201.11:5443 it works fine and without a problem.
Once we want to connect the cluster from an external connection like https://80.158.17.199:5443 gitlab throws an SSL Verify error:
{
"severity": "ERROR",
"time": "2020-07-27T05:46:22.500Z",
"correlation_id": "AjImNPsdq23",
"exception": "Kubeclient::HttpError",
"status_code": null,
"namespace": "gitlab-managed-apps",
"class_name": "Gitlab::Kubernetes::Namespace",
"event": "failed_to_create_namespace",
"message": "SSL_connect returned=1 errno=0 state=error: certificate verify failed (unspecified certificate verification error)"
}
Using the regular kubectl from the same machine with the same CA File, Service Token and external IP also works without problems.
What is the expected correct behavior?
It should connect and not throw SSL Verify Errors
Relevant logs and/or screenshots
I've also added the Cert to /etc/gitlab/trusted-certs/kubernetes-cert.pem and did gitlab-ctl reconfigure, which created a symlink to /opt/gitlab/embedded/ssl/certs/cfe984e6.0 as expected.
Testing the embedded SSL Client also shows no cert errors:
root@ecs-git01:~# echo | /opt/gitlab/embedded/bin/openssl s_client -connect 80.158.17.199:5443
CONNECTED(00000003)
Can't use SSL_get_servername
depth=1 O = CCE Technologies
verify return:1
depth=0 O = CCE Technologies, CN = system:kube-apiserver
verify return:1
---
Certificate chain
0 s:O = CCE Technologies, CN = system:kube-apiserver
i:O = CCE Technologies
---
Server certificate
-----BEGIN CERTIFICATE-----
MIID2DCCAsCgAwIBAgIILrSds4FCKWMwDQYJKoZIhvcNAQELBQAwGzEZMBcGA1UE
ChMQQ0NFIFRlY2hub2xvZ2llczAgFw0yMDA3MTcwNjQ2MjRaGA8yMDUwMDcxMDA2
NDYyNVowOzEZMBcGA1UEChMQQ0NFIFRlY2hub2xvZ2llczEeMBwGA1UEAxMVc3lz
dGVtOmt1YmUtYXBpc2VydmVyMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKC
AQEAltRugTCX1I16g5B3N64od6kDck1aRKsMpEm57EHSQrtTW+l55RdXx49seRbE
2xMNxkVIJM+3tMQCkT4cLbviioPOTQ3NuFyPjD57XeUVZVYZPEBHqW/33gwrD945
KvZTGnUu4JJn+elO9GGYSLumAdLbDVqpYbOayiC1pn/BYMIg/b+p2CYXcAeIDsJ2
Q2J8VLbtTDEECNOZbcP4NRnrtgxcbKnhiXs6ri/d4/yqTDLDyZICbBYBxCMdy1in
HRG6hHknitPtjHiSd7hm1gk9qsJh5orcaO7WRclSQR7UNq1IZxE5R9w2wAbkcytG
dxALMC+4JhXVr71XaFiKc+PEZQIDAQABo4H9MIH6MA4GA1UdDwEB/wQEAwIFoDAd
BgNVHSUEFjAUBggrBgEFBQcDAgYIKwYBBQUHAwEwgcgGA1UdEQSBwDCBvYIWa3Vi
ZXJuZXRlcy5kZWZhdWx0LnN2Y4Ika3ViZXJuZXRlcy5kZWZhdWx0LnN2Yy5jbHVz
dGVyLmxvY2FsghJrdWJlcm5ldGVzLmRlZmF1bHSCCmt1YmVybmV0ZXOCJ2lzdGlv
LXNpZGVjYXItaW5qZWN0b3IuaXN0aW8tc3lzdGVtLnN2Y4cQ/QASNAAAAAAAAAAA
AAAAAYcEfwAAAYcECvcAAocECvcAAYcEZFPzm4cECgCKYYcEwKjKVjANBgkqhkiG
9w0BAQsFAAOCAQEAbuODIhu023MUInB6TpcrTLurkVIXnxvZTWXxPl9S9ErlIN45
p8q8MxZkDNsIrZlnNHK3z62IEfvDQ5v8lwdK/ZCt65EL0tcnsOwnpGvC9o0e3MBg
GaGzExNZvB7Rv5SYtWD0r3e1yPX9Syq/NpFtz4QMDXW6ISd1CRp2IePVg4kqxe0S
WJ/4OVp2ea1AEIn+Yyd9LkJD7mMysHV7cnxG0r6HWMvUdH+gZ+ggr33qtXRQ1FeT
ICaP/T5TavGAdYSDMNsRMRJvHTwh8kGLFSSl/9tMvZgnO74KzdMiy1ZDQsQHLaG8
GncEQGyQiGYJyfcalf46HDKtXGDVM3/j0rRWLA==
-----END CERTIFICATE-----
subject=O = CCE Technologies, CN = system:kube-apiserver
issuer=O = CCE Technologies
---
Acceptable client certificate CA names
O = CCE Technologies
O = CCE Technologies
Client Certificate Types: RSA sign, ECDSA sign
Requested Signature Algorithms: RSA+SHA256:ECDSA+SHA256:RSA+SHA384:ECDSA+SHA384:RSA+SHA512:ECDSA+SHA512:RSA+SHA1:ECDSA+SHA1
Shared Requested Signature Algorithms: RSA+SHA256:ECDSA+SHA256:RSA+SHA384:ECDSA+SHA384:RSA+SHA512:ECDSA+SHA512:RSA+SHA1:ECDSA+SHA1
Peer signing digest: SHA256
Peer signature type: RSA
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 1655 bytes and written 398 bytes
Verification: OK
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES128-GCM-SHA256
Session-ID: 91A7367007B26F314A63AD45C7F5434B87F9AB6ECAC7A520E4AA2FACDB8FCEC4
Session-ID-ctx:
Master-Key: 173E0A6DD1FC16642230A7E08DE2A7D84D84D225AE8106E6B069BA42A4931FFFCCE4F244DEE3D025EBA150D23F3A1C65
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket:
0000 - b2 8e 5e 15 10 de 0b 09-5e 57 39 1f 3e 19 92 ec ..^.....^W9.>...
0010 - db 26 80 2c 4b bd ba f1-63 45 dd 5d fd 68 97 06 .&.,K...cE.].h..
0020 - 66 8e 7c 8a c7 3e a7 9c-e4 8d 98 7e f7 e4 c4 c3 f.|..>.....~....
0030 - 57 33 71 3f 1b cd 2f 24-62 7f 1e 26 8d c9 3e bd W3q?../$b..&..>.
0040 - fb fe 03 bf 17 11 c5 1e-f9 65 4a 6a dd 37 30 74 .........eJj.70t
0050 - 3e 2a a9 b3 22 ca 79 af-42 ee 2f 87 33 7d 01 96 >*..".y.B./.3}..
0060 - 54 6b 8e 20 25 14 2c 7f-2a b7 b1 76 cd 06 81 c1 Tk. %.,.*..v....
0070 - a6 bd f7 d1 c0 fd 16 4d- .......M
Start Time: 1595829513
Timeout : 7200 (sec)
Verify return code: 0 (ok)
Extended master secret: no
---
DONE
Output of checks
This bug happens on GitLab.com as well as on our own self-hosted instance
Results of GitLab application Check
Not Sure if this is relevant as this is also happening on gitlab.com
System information
System: Debian 9.13
Current User: git
Using RVM: no
Ruby Version: 2.6.3p62
Gem Version: 2.7.9
Bundler Version:1.17.3
Rake Version: 12.3.2
Redis Version: 3.2.12
Git Version: 2.21.0
Sidekiq Version:5.2.7
Go Version: unknown
GitLab information
Version: 12.1.6
Revision: 4016bcac51d
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: PostgreSQL
DB Version: 10.7
URL: https://git.sec.test.tsi-dev.otc-service.com
HTTP Clone URL: https://git.sec.test.tsi-dev.otc-service.com/some-group/some-project.git
SSH Clone URL: git@git-ssh.sec.test.tsi-dev.otc-service.com:some-group/some-project.git
Using LDAP: yes
Using Omniauth: yes
Omniauth Providers: saml
GitLab Shell
Version: 9.3.0
Repository storage paths:
- default: /srv/gitlab/repositories
GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell
Git: /opt/gitlab/embedded/bin/git
root@ecs-git01:~# gitlab-rake gitlab:check SANITIZE=true
Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 9.3.0 ? ... OK (9.3.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Redis available via internal API: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes
Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... Server: ldapmain
LDAP authentication... Success
LDAP users with access to your GitLab server (only showing the first 100 results)
User output sanitized. Found 100 users of 100 limit.
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... skipped (no tmp uploads folder yet)
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
5/3 ... yes
8/4 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.5.3 ? ... yes (2.6.3)
Git version >= 2.21.0 ? ... yes (2.21.0)
Git user has default SSH configuration? ... yes
Active users: ... 7
Checking GitLab App ... Finished
Checking GitLab subtasks ... FinishedPossible fixes
Not quite sure, somehow the kubernetes integration is not using the embedded ssl but some ruby lib that might not be checking the cert correctly.