SSL problem with packages.gitlab.com on Ubuntu 18.04 LTS: apt-get update: certificate not trusted since GitLab 12.8.1
Having auto-updates enabled in Ubuntu 18.04 LTS, i was surprised to find our gitlab installation still using 12.8.1 when i logged in as admin today.
Investigation showed this:
# apt-get update
...
Get:4 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
Ign:5 https://packages.gitlab.com/gitlab/gitlab-ce/ubuntu bionic InRelease
Err:6 https://packages.gitlab.com/gitlab/gitlab-ce/ubuntu bionic Release
Certificate verification failed: The certificate is NOT trusted. The certificate issuer is unknown. Could not handshake: Error in the certificate verification. [IP: 54.153.54.194 443]
Reading package lists... Done
E: The repository 'https://packages.gitlab.com/gitlab/gitlab-ce/ubuntu bionic Release' no longer has a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.# curl -L https://packages.gitlab.com/gitlab/gitlab-ce/ubuntu/Release
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.# apt-key list
/etc/apt/trusted.gpg
--------------------
pub rsa4096 2015-04-17 [SC] [expires: 2020-04-15]
1A4C 919D B987 D435 9396 38B9 1421 9A96 E15E 78F4
uid [ unknown] GitLab B.V. (package repository signing key) <packages@gitlab.com># openssl s_client -showcerts -connect packages.gitlab.com:443
CONNECTED(00000005)
depth=1 C = GB, ST = Greater Manchester, L = Salford, O = Sectigo Limited, CN = Sectigo RSA Domain Validation Secure Server CA
verify error:num=20:unable to get local issuer certificate
---
Certificate chain
0 s:CN = packages.gitlab.com
i:C = GB, ST = Greater Manchester, L = Salford, O = Sectigo Limited, CN = Sectigo RSA Domain Validation Secure Server CA
-----BEGIN CERTIFICATE-----
MIIFzjCCBLagAwIBAgIQf1AZtnAiZu9JPXtf3RB5JDANBgkqhkiG9w0BAQsFADCB
jzELMAkGA1UEBhMCR0IxGzAZBgNVBAgTEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4G
A1UEBxMHU2FsZm9yZDEYMBYGA1UEChMPU2VjdGlnbyBMaW1pdGVkMTcwNQYDVQQD
Ey5TZWN0aWdvIFJTQSBEb21haW4gVmFsaWRhdGlvbiBTZWN1cmUgU2VydmVyIENB
MB4XDTE5MTIyOTAwMDAwMFoXDTIxMDIyNjIzNTk1OVowHjEcMBoGA1UEAxMTcGFj
a2FnZXMuZ2l0bGFiLmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
ALCRe8seTMi9oKbIWPfqDbsfAmg4bAvqmHsZQsWG1hsz7B+7+NcQ2PE4ccBB2SNg
Kry60PhxHCeq9ZWjZKBMrCchKTAU0NtfkxeQVA6QzJYKZvUQaZ3YPEhpo5PoYYxs
NbX6+XasioglArR3y4GNHv30VF2AGO/sxngyjJGWtx0Lo3SyYVmG+U3tCNZSXVtv
saMr2KjJENZ6dFOc6trumjZu787101dKO9aiFS2E25EHZTIiA/6MJ8VKuUt0BgPq
XzGqgIIF5TRKYOp+BOgvnSk41AyboJ6CT9VFXHOlO9l2tT5HbRpUj9mQ0g+zNfKA
g4VkcFKJtztUwPvTYFLbfmECAwEAAaOCApQwggKQMB8GA1UdIwQYMBaAFI2MXsRU
rYrhd+mb+ZsF4bgBjWHhMB0GA1UdDgQWBBQ0zJfJlm4L+igOWq49rwyAzP/h6DAO
BgNVHQ8BAf8EBAMCBaAwDAYDVR0TAQH/BAIwADAdBgNVHSUEFjAUBggrBgEFBQcD
AQYIKwYBBQUHAwIwSQYDVR0gBEIwQDA0BgsrBgEEAbIxAQICBzAlMCMGCCsGAQUF
BwIBFhdodHRwczovL3NlY3RpZ28uY29tL0NQUzAIBgZngQwBAgEwgYQGCCsGAQUF
BwEBBHgwdjBPBggrBgEFBQcwAoZDaHR0cDovL2NydC5zZWN0aWdvLmNvbS9TZWN0
aWdvUlNBRG9tYWluVmFsaWRhdGlvblNlY3VyZVNlcnZlckNBLmNydDAjBggrBgEF
BQcwAYYXaHR0cDovL29jc3Auc2VjdGlnby5jb20wNwYDVR0RBDAwLoITcGFja2Fn
ZXMuZ2l0bGFiLmNvbYIXd3d3LnBhY2thZ2VzLmdpdGxhYi5jb20wggEEBgorBgEE
AdZ5AgQCBIH1BIHyAPAAdgB9PvL4j/+IVWgkwsDKnlKJeSvFDngJfy5ql2iZfiLw
1wAAAW9PBZGdAAAEAwBHMEUCIQDbghim8eBUPPMUnTlbGnmrBowwUzQ+rkIEvJUq
8znnwQIgLsYZR4aoyiYbipjt5dvsl1Dd6R4Z729d2edAf49s6tYAdgBElGUusO7O
r8RAB9io/ijA2uaCvtjLMbU/0zOWtbaBqAAAAW9PBZFeAAAEAwBHMEUCIQDXtljG
7s58KeH9c1fkNYHcOyfPq8dZgB9tAgriMIibuQIgTM5rhLYZjZVaTKVPjdYs2s38
zYFrGnwZbd1lNbQq4JowDQYJKoZIhvcNAQELBQADggEBANAzweRsM71mKF6jb5mN
76tE4582LEuJOWJLD4DV5/ZZYlDksRnC1wlqDTuKWCxpB417qlM6/Ay0zNOUxr5l
cRc51STa5h+tijiOWKeLmQib8Ax11NX15IZmCTyOtlKlcPbgcfv0qYxbrijZ3azq
yJNJOw+TgMevmbBVUbPmNJXRqTJvSOdIJze4Wu/XmWLvaUgpJ8FVi/N7IiumCGUa
1O/gayKHzVmr2c4g3M6UF9wNh26JKxFpSoFtyrdev+I1d62YXDFg+hNTUBMZ/npA
W3O3RjNKmYyfAGMTk/TKhsJ3Jie9/XNoJ195d8irVp/XVqY6VeHikuAV7c/3Yoa+
XGw=
-----END CERTIFICATE-----
1 s:C = GB, ST = Greater Manchester, L = Salford, O = Sectigo Limited, CN = Sectigo RSA Domain Validation Secure Server CA
i:C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, CN = USERTrust RSA Certification Authority
-----BEGIN CERTIFICATE-----
MIIGEzCCA/ugAwIBAgIQfVtRJrR2uhHbdBYLvFMNpzANBgkqhkiG9w0BAQwFADCB
iDELMAkGA1UEBhMCVVMxEzARBgNVBAgTCk5ldyBKZXJzZXkxFDASBgNVBAcTC0pl
cnNleSBDaXR5MR4wHAYDVQQKExVUaGUgVVNFUlRSVVNUIE5ldHdvcmsxLjAsBgNV
BAMTJVVTRVJUcnVzdCBSU0EgQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkwHhcNMTgx
MTAyMDAwMDAwWhcNMzAxMjMxMjM1OTU5WjCBjzELMAkGA1UEBhMCR0IxGzAZBgNV
BAgTEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4GA1UEBxMHU2FsZm9yZDEYMBYGA1UE
ChMPU2VjdGlnbyBMaW1pdGVkMTcwNQYDVQQDEy5TZWN0aWdvIFJTQSBEb21haW4g
VmFsaWRhdGlvbiBTZWN1cmUgU2VydmVyIENBMIIBIjANBgkqhkiG9w0BAQEFAAOC
AQ8AMIIBCgKCAQEA1nMz1tc8INAA0hdFuNY+B6I/x0HuMjDJsGz99J/LEpgPLT+N
TQEMgg8Xf2Iu6bhIefsWg06t1zIlk7cHv7lQP6lMw0Aq6Tn/2YHKHxYyQdqAJrkj
eocgHuP/IJo8lURvh3UGkEC0MpMWCRAIIz7S3YcPb11RFGoKacVPAXJpz9OTTG0E
oKMbgn6xmrntxZ7FN3ifmgg0+1YuWMQJDgZkW7w33PGfKGioVrCSo1yfu4iYCBsk
Haswha6vsC6eep3BwEIc4gLw6uBK0u+QDrTBQBbwb4VCSmT3pDCg/r8uoydajotY
uK3DGReEY+1vVv2Dy2A0xHS+5p3b4eTlygxfFQIDAQABo4IBbjCCAWowHwYDVR0j
BBgwFoAUU3m/WqorSs9UgOHYm8Cd8rIDZsswHQYDVR0OBBYEFI2MXsRUrYrhd+mb
+ZsF4bgBjWHhMA4GA1UdDwEB/wQEAwIBhjASBgNVHRMBAf8ECDAGAQH/AgEAMB0G
A1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAbBgNVHSAEFDASMAYGBFUdIAAw
CAYGZ4EMAQIBMFAGA1UdHwRJMEcwRaBDoEGGP2h0dHA6Ly9jcmwudXNlcnRydXN0
LmNvbS9VU0VSVHJ1c3RSU0FDZXJ0aWZpY2F0aW9uQXV0aG9yaXR5LmNybDB2Bggr
BgEFBQcBAQRqMGgwPwYIKwYBBQUHMAKGM2h0dHA6Ly9jcnQudXNlcnRydXN0LmNv
bS9VU0VSVHJ1c3RSU0FBZGRUcnVzdENBLmNydDAlBggrBgEFBQcwAYYZaHR0cDov
L29jc3AudXNlcnRydXN0LmNvbTANBgkqhkiG9w0BAQwFAAOCAgEAMr9hvQ5Iw0/H
ukdN+Jx4GQHcEx2Ab/zDcLRSmjEzmldS+zGea6TvVKqJjUAXaPgREHzSyrHxVYbH
7rM2kYb2OVG/Rr8PoLq0935JxCo2F57kaDl6r5ROVm+yezu/Coa9zcV3HAO4OLGi
H19+24rcRki2aArPsrW04jTkZ6k4Zgle0rj8nSg6F0AnwnJOKf0hPHzPE/uWLMUx
RP0T7dWbqWlod3zu4f+k+TY4CFM5ooQ0nBnzvg6s1SQ36yOoeNDT5++SR2RiOSLv
xvcRviKFxmZEJCaOEDKNyJOuB56DPi/Z+fVGjmO+wea03KbNIaiGCpXZLoUmGv38
sbZXQm2V0TP2ORQGgkE49Y9Y3IBbpNV9lXj9p5v//cWoaasm56ekBYdbqbe4oyAL
l6lFhd2zi+WJN44pDfwGF/Y4QA5C5BIG+3vzxhFoYt/jmPQT2BVPi7Fp2RBgvGQq
6jG35LWjOhSbJuMLe/0CjraZwTiXWTb2qHSihrZe68Zk6s+go/lunrotEbaGmAhY
LcmsJWTyXnW0OMGuf1pGg+pRyrbxmRE1a6Vqe8YAsOf4vmSyrcjC8azjUeqkk+B5
yOGBQMkKW+ESPMFgKuOXwIlCypTPRpgSabuY0MLTDXJLR27lk8QyKGOHQ+SwMj4K
00u/I5sUKUErmgQfky3xxzlIPK1aEn8=
-----END CERTIFICATE-----
---
Server certificate
subject=CN = packages.gitlab.com
issuer=C = GB, ST = Greater Manchester, L = Salford, O = Sectigo Limited, CN = Sectigo RSA Domain Validation Secure Server CA
---
No client certificate CA names sent
Peer signing digest: SHA512
Peer signature type: RSA
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 3742 bytes and written 447 bytes
Verification error: unable to get local issuer certificate
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES128-GCM-SHA256
Session-ID: BBE75F0D58EA39C0836FA2D7A43AD9D1BADC2D81F5099EFFB21CC9F3843486C2
Session-ID-ctx:
Master-Key: 5F4B0AB5E7AD835384EAEA82B8BAA0F196187D3875FF306FF028CE64F9778F3D73A0092D5787FA98A057A893C8F24F39
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket lifetime hint: 300 (seconds)
TLS session ticket:
0000 - 79 a1 9a 1a e1 f8 b1 c4-e5 cb b9 02 7d 71 98 fd y...........}q..
0010 - 12 b4 31 10 94 5d 3d 50-10 0d c4 7e be 89 59 a0 ..1..]=P...~..Y.
0020 - e0 43 2b 60 44 ed a9 e6-dd fb 75 8d 82 ad 9e 20 .C+`D.....u....
0030 - dc 2e 62 ed 1a 4b df dd-6f e4 59 f1 00 99 dd 68 ..b..K..o.Y....h
0040 - 78 10 a3 8f 7c fa 0c 52-50 38 d9 95 d7 bb be 67 x...|..RP8.....g
0050 - e7 3f 6f e5 bf 28 4f ee-b4 62 73 3b 04 0c e0 1d .?o..(O..bs;....
0060 - d5 e7 a5 32 4e 33 ec 63-1e 94 56 82 b6 8e 2e c9 ...2N3.c..V.....
0070 - 52 b9 2c 7d 20 ee 0c 4d-52 95 0c 71 fa 8f d9 c6 R.,} ..MR..q....
0080 - c3 03 76 11 3a e1 66 0f-be 0c da 99 f2 1b a0 2e ..v.:.f.........
0090 - 39 ac 79 fc c6 47 99 22-49 33 37 43 24 36 43 e6 9.y..G."I37C$6C.
00a0 - b8 47 ce 1b 64 58 1b dd-38 08 c8 60 d0 01 7f 42 .G..dX..8..`...B
00b0 - ef 56 e6 43 c1 eb 27 f9-93 30 c1 b6 47 7e ce b1 .V.C..'..0..G~..
Start Time: 1583749505
Timeout : 7200 (sec)
Verify return code: 20 (unable to get local issuer certificate)
Extended master secret: no
---
DONEAs you can see, the Ubuntu 18.04 LTS installation doesn't seem to accept the root (USERTrust CA) certificate...
I'm marking this as confidential for now, as it means that your security release 12.8.2 might not have been applied automatically to a large number of instances!