Self-hosted cloud activation fails when SSL Deep Inspection is used within a network

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Summary

Customer Support has received two tickets (one, two) where each customer uses a network appliance in their corporate network with some form of SSL Deep Inspection enabled.

SSL Deep Inspection has the appliance impersonate the HTTPS session, leading to the certificate chain being altered from a host perspective. As seen from the two customers linked above, the result of querying for SSL issuers reports something similar to the following:

# Extract specific information from the certificate being served by customers.gitlab.com
echo | openssl s_client -connect customers.gitlab.com:443 -servername customers.gitlab.com 2</dev/null | openssl x509 -subject -issuer -dates -noout

# This is an example of output seen by one customer.
subject=C = US, ST = California, L = San Francisco, O = "Cloudflare, Inc.", CN = gitlab.com
issuer=C = US, ST = California, L = Sunnyvale, O = Fortinet, OU = Certificate Authority, CN = <serial>, emailAddress = support@fortinet.com
notBefore=Nov  1 00:00:00 2022 GMT
notAfter=Jan 30 23:59:59 2023 GMT

# This is a different customer, with a different network appliance (sanitised to "example")
subject=C = US, ST = California, L = San Francisco, O = "Cloudflare, Inc.", CN = gitlab.com
issuer=C = US, ST = <example>, L = <example>, O = <example>, OU = <example>, CN = <example>, emailAddress = enterprise_network_perimeter@<example.com>
notBefore=Sep 2 00:00:00 2022 GMT
notAfter=Dec 1 23:59:59 2022 GMT

The expected output is Cloudflare is the issuer of the certificate:

subject=C = US, ST = California, L = San Francisco, O = "Cloudflare, Inc.", CN = gitlab.com
issuer=C = US, O = "Cloudflare, Inc.", CN = Cloudflare Inc ECC CA-3
notBefore=Nov  1 00:00:00 2022 GMT
notAfter=Jan 30 23:59:59 2023 GMT

When a customer attempts to activate a Cloud Activation License, it will not succeed, and an error message will be presented in GraphQL output in Developer Tools of the browser. Here's an example:

[{"data":{"gitlabSubscriptionActivate":{"errors":["SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)"],"license":null,"__typename":"GitlabSubscriptionActivatePayload"}}}]

This seems to occur because the GitLab instance must trust the injected issuer. This presents a question of how we should proceed:

Don't change GitLab behaviour

This option would not change how we check against the SSL certificate chain. We could improve the user process by:

  • Returning a specific error message to the UI, indicating this could be a certificate chain problem
  • Add further documentation that if you use a network appliance with SSL Deep Inspection, you must add the custom CA issuer to the /etc/gitlab/trusted-certs folder. This looks to be a recommended step by network appliance vendors, as FortiGate indicates (from the perspective of a user's browser) that:

Because there is no Fortinet_CA_SSL in the browser trusted CA list, the browser displays an untrusted certificate warning when it receives a FortiGate re-signed server certificate. To stop the warning messages, trust the FortiGate-trusted CA Fortinet_CA_SSL and import it into your browser.

Change GitLab behaviour

  • For activation purposes, ignore the "self-signed" certificate (the issuer)? This could present a security risk as it bypasses the certificate trust system.

Steps to reproduce

This is currently difficult to do so without a readily available network appliance with SSL Deep Inspection. @benjaminking is investigating ways in which to gain temporary access to a demo appliance to test this further.

What is the current bug behavior?

Cloud Activation fails, resulting in the user being unable to activate their GitLab instance.

What is the expected correct behavior?

Cloud Activation should complete successfully, pending how we may want to proceed with the issue.

Edited by 🤖 GitLab Bot 🤖