<!--- Please read this! Before opening a new issue, make sure to search for keywords in the issues filtered by the "regression" or "bug" label. For the Community Edition issue tracker: - https://gitlab.com/gitlab-org/gitlab-ce/issues?label_name%5B%5D=regression - https://gitlab.com/gitlab-org/gitlab-ce/issues?label_name%5B%5D=bug For the Enterprise Edition issue tracker: - https://gitlab.com/gitlab-org/gitlab-ee/issues?label_name%5B%5D=regression - https://gitlab.com/gitlab-org/gitlab-ee/issues?label_name%5B%5D=bug and verify the issue you're about to submit isn't a duplicate. ---> ### Summary We are trying to enable Webterminal for our Gitlab-CE instance. Direct access to the nodes with the gitlab-runners is working. The Webterminal connects and we can access the containers. ``` [...] Terminal is connected, will time out in 30m0s... Terminal disconnected Job succeeded ``` The gitlab-runners are running in our Kubernetes cluster (which is otherwise independent of Gitlab), access to the runners (for now) is made available via Ingress L4 load balancing. To make sure we can always access our runners, even in case of a node shutdown, we use our reverse proxy for load balancing and SSL termination. We have a valid wildcard Let's Encrypt certificate installed: ``` # curl -v https://gitlab-runner.ourcompany.com [...] * Server certificate: * subject: CN=ourcompany.com * start date: Mar 22 08:51:56 2019 GMT * expire date: Jun 20 08:51:56 2019 GMT * subjectAltName: host "gitlab-runner.ourcompany.com" matched cert's "*.ourcompany.com" * issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3 * SSL certificate verify ok. [...] < 404 page not found * Connection #0 to host gitlab-runner.ourcompany.com left intact ``` However, when we tell Gitlab to connect to the webterminal via the reverse proxy, it fails with the following entry in gitlab-workhorse logs: `2019-03-26_10:28:08.94238 time="2019-03-26T11:28:08+01:00" level=error msg=error correlation_id=vQ8Feb9qu76 error="x509: certificate signed by unknown authority" method=GET uri=/group/ci-tests/-/jobs/8868/terminal.ws` It seems the Let's Encrypt CA is not part of the cacert.pem file: ``` # openssl verify -CAfile /opt/gitlab/embedded/ssl/certs/cacert.pem our-lets-encrypt-cert.pem CN = ourcompany.com error 20 at 0 depth lookup: unable to get local issuer certificate error our-lets-encrypt-cert.pem: verification failed ``` Following the steps in https://docs.gitlab.com/omnibus/settings/ssl.html#connecting-to-external-resources we added the Let's Encrypt CA chain: ``` # ls /opt/gitlab/embedded/ssl/certs 2e5ac55d.0 4f06f81d.0 cacert.pem README # openssl verify -CApath /opt/gitlab/embedded/ssl/certs/ -show_chain our-lets-encrypt-cert.pem our-lets-encrypt-cert.pem: OK Chain: depth=0: CN = ourcompany.com (untrusted) depth=1: C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3 depth=2: O = Digital Signature Trust Co., CN = DST Root CA X3 ``` Sadly, this does not change anything. Are we missing something? We are not very familiar with the internal works of gitlab, so it might be a user error. We tried several things: * adding the certificates directly to the cacert.pem file * setting the SSL_CERT_DIR env variable for workhorse to /etc/ssl/certs where all necessary CA certs are contained. * testing the connection with `gitlab-rails console` and `Net::HTTP.get(URI(...))`, no errors there. ### Steps to reproduce 1. Setup gitlab-runner with session_server enabled. 2. Make sure it works with direct connecting gitlab-workhorse -> gitlab-runner 3. Setup a reverse proxy with a valid Let's Encrypt certificate 4. Copy the certificate chain to `/etc/gitlab/trusted-certs` and run `gitlab-ctl reconfigure` 4. Update the runner config such that gitlab-workhorse connects to the reverse proxy for the webterminal ### What is the current *bug* behavior? It seems the added Let's Encrypt CA is not loaded und thus the certificate cannot be validated. ### What is the expected *correct* behavior? The presented certificate of the reverse proxy is validated against the provided Let's Encrypt CA chain and the connection can be established. ### Relevant logs and/or screenshots See summary. ### Output of checks (If you are reporting a bug on GitLab.com, write: This bug happens on GitLab.com) #### Results of GitLab environment info <details> <summary>Expand for output related to GitLab environment info</summary> <pre> System information System: Ubuntu 18.04 Current User: git Using RVM: no Ruby Version: 2.5.3p105 Gem Version: 2.7.6 Bundler Version:1.16.6 Rake Version: 12.3.2 Redis Version: 3.2.12 Git Version: 2.18.1 Sidekiq Version:5.2.5 Go Version: unknown GitLab information Version: 11.9.0 Revision: a47124c7852 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: postgresql URL: https://gitlab.mycompany.com HTTP Clone URL: https://gitlab.mycompany.com/some-group/some-project.git SSH Clone URL: ssh://git@gitlab.mycompany.com:1848/some-group/some-project.git Using LDAP: yes Using Omniauth: no GitLab Shell Version: 8.7.1 Repository storage paths: - default: /projects/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git </pre> </details> #### Results of GitLab application Check <details> <summary>Expand for output related to the GitLab application check</summary> <pre> (For installations with omnibus-gitlab package run and paste the output of: `sudo gitlab-rake gitlab:check SANITIZE=true`) (For installations from source run and paste the output of: `sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true`) (we will only investigate if the tests are passing) </pre> </details> ### Details of package version <details> <summary>Provide the package version installation details</summary> <pre> # dpkg-query -l "gitlab-*" Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-===============================================-============================-============================-=================================================================================================== ii gitlab-ce 11.9.1-ce.0 amd64 GitLab Community Edition (including NGINX, Postgres, Redis) un gitlab-ee <none> <none> (no description available) </pre> </details> ### Environment details * Operating System: `Ubuntu 18.04.1 LTS` * Installation Target, remove incorrect values: * VM: `KVM` * Installation Type, remove incorrect values: * Updated: Always kept up-to-date * Is there any other software running on the machine: `besides orchestration, no` * Is this a single or multiple node installation? `single` * Resources * CPU: `4` * Memory total: `6G` ### Configuration details <details> <summary>Provide the relevant sections of `/etc/gitlab/gitlab.rb`</summary> <pre> external_url 'https://gitlab.mycompany.com' git_data_dirs({"default"=>{"path"=>"/projects/gitlab/git-data"}}) gitlab_rails['backup_archive_permissions'] = 511 gitlab_rails['backup_keep_time'] = 1209600 gitlab_rails['backup_path'] = "/storage/backup/gitlab" gitlab_rails['db_adapter'] = "postgresql" gitlab_rails['db_database'] = "gitlab" gitlab_rails['db_host'] = "xxx" gitlab_rails['db_password'] = "xxx" gitlab_rails['db_pool'] = 10 gitlab_rails['db_username'] = "gitlab" gitlab_rails['gitlab_default_can_create_group'] = false gitlab_rails['gitlab_default_projects_features_builds'] = false gitlab_rails['gitlab_default_projects_features_issues'] = true gitlab_rails['gitlab_default_projects_features_merge_requests'] = true gitlab_rails['gitlab_default_projects_features_snippets'] = false gitlab_rails['gitlab_default_projects_features_visibility_level'] = "private" gitlab_rails['gitlab_default_projects_features_wiki'] = false gitlab_rails['gitlab_email_from'] = "gitlab@mycompany.com" gitlab_rails['gitlab_restricted_visibility_levels'] = ["public"] gitlab_rails['gitlab_shell_ssh_port'] = 1234 gitlab_rails['gitlab_ssh_host'] = "gitlab.mycompany.com" gitlab_rails['gitlab_username_changing_enabled'] = false gitlab_rails['incoming_email_address'] = "gitlab+%{key}@service.mycompany.com" gitlab_rails['incoming_email_email'] = "gitlab@service.mycompany.com" gitlab_rails['incoming_email_enabled'] = true gitlab_rails['incoming_email_host'] = "mail.mycompany.com" gitlab_rails['incoming_email_mailbox_name'] = "inbox" gitlab_rails['incoming_email_password'] = "xxx" gitlab_rails['incoming_email_port'] = 143 gitlab_rails['incoming_email_ssl'] = false gitlab_rails['incoming_email_start_tls'] = true gitlab_rails['ldap_enabled'] = true gitlab_rails['ldap_servers'] = xxx gitlab_rails['lfs_enabled'] = true gitlab_rails['lfs_storage_path'] = "/projects/gitlab/git-lfs" gitlab_rails['manage_backup_path'] = false gitlab_rails['omniauth_enabled'] = false gitlab_rails['time_zone'] = "xxx" gitlab_rails['trusted_proxies'] = ["the proxy in question is contained in this list"] sidekiq['concurrency'] = 5 postgresql['enable'] = false nginx['listen_https'] = false nginx['listen_port'] = 80 nginx['redirect_http_to_https'] = false logging['svlogd_filter'] = "gzip" logging['svlogd_num'] = 30 logging['svlogd_size'] = 209715200 logging['svlogd_timeout'] = 86400 prometheus_monitoring['enable'] = true prometheus['enable'] = false node_exporter['enable'] = false redis_exporter['enable'] = true redis_exporter['listen_address'] = "0.0.0.0:9121" postgres_exporter['enable'] = false gitlab_monitor['enable'] = true gitlab_monitor['listen_address'] = "0.0.0.0" gitlab_monitor['listen_port'] = 9168 pages_external_url 'http://pages.mycompany.com' gitlab_pages['access_control'] = true pages_nginx['redirect_http_to_https'] = false gitaly['prometheus_listen_addr'] = "0.0.0.0:9175" </pre></details> ### Possible fixes Sadly, none.