First of all, I don't think this is a bug or a problem in GitLab. It is most likely a misconfiguration in my system or something I am doing wrong. I would just like some guidance on what to look for. My GitLab is running in Proxmox:LXC:Ubuntu 18.4 LTS:GitLab-ce 11.11.3 Omnibus. I setup Kubernetes in Windows 10:VMware Player:Linux Mint 18.2 using [MicroK8s](https://microk8s.io/) as it is recommended for systems with less than 32GB of RAM. I assigned my Mint VM 8GB of RAM. There are two systems I am using, both are resolvable in my LAN by their hostnames __GitLab.BigHouse__ and __Asus-ROG-VM.BigHouse__. ``` sudo -u git -H gitlab-rake gitlab:env:info System information System: Ubuntu 18.04 Current User: git Using RVM: no Ruby Version: 2.5.3p105 Gem Version: 2.7.9 Bundler Version:1.17.3 Rake Version: 12.3.2 Redis Version: 3.2.12 Git Version: 2.21.0 Sidekiq Version:5.2.7 Go Version: unknown GitLab information Version: 11.11.3 Revision: e3eeb779d72 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 9.6.11 URL: http://GitLab.BigHouse HTTP Clone URL: http://GitLab.BigHouse/some-group/some-project.git SSH Clone URL: git@GitLab.BigHouse:some-group/some-project.git Using LDAP: no Using Omniauth: yes Omniauth Providers: GitLab Shell Version: 9.1.0 Repository storage paths: - default: /home/git/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git ``` What I am trying to do is connect my GitLab CI/CD to my Kubernetes in the VMware Mint. I have been using these pages to guide me. - [Connecting GitLab with a Kubernetes cluster](https://docs.gitlab.com/ee/user/project/clusters/) - [Setup a runner for an existing cluster](https://gitlab.hubot.com.br/help/user/project/clusters/index.md) The Microk8s kube-apiserver is setup by default with a self-signed certificate with the CN=127.0.0.1 so it is only valid for local API calls. I used curl from my GitLab.BigHouse to connect to the API at __https://Asus-ROG-VM.BigHouse:16443__ and got an SSL error as expected. I used this guide to create a self-signed certificate with CN=Asus-ROG-VM.BigHouse [Creating TLS certificates using Kubernetes API](https://db-blog.web.cern.ch/blog/lukas-gedvilas/2018-02-creating-tls-certificates-using-kubernetes-api) I set the new certificate as arguments for kube-apiserver as __tls-cert-file__ and __tls-private-key-file__ in /var/snap/microk8s/current/args/kube-apiserver. I then restarted microk8s so the new certificate would take effect. I ran a few kubectl commands, it seems to still able to interact with the API locally. On GitLab.BigHouse I manually added the PEM to the OS using these commands. ``` echo | openssl s_client -showcerts -connect Asus-ROG-VM.BigHouse:16443 2>/dev/null | openssl x509 -outform PEM > /usr/local/share/ca-certificates/kubernetes.Asus-ROG-VM.BigHouse.crt update-ca-certificates ``` ``` curl https://Asus-ROG-VM.BigHouse:16443 { "kind": "Status", "apiVersion": "v1", "metadata": { }, "status": "Failure", "message": "Unauthorized", "reason": "Unauthorized", "code": 401 } ``` This returns a 401 because I am not using a token, but it shows the SSL issue has been resolved, at least for curl it has. I added the new PEM to the GitLab __Kubernetes cluster details__ under __CA Certificate__. The token and URL had already been set. Then I go to the GitLab page to configure the runner and click on the button to install Helm, it fails with __Something went wrong while installing Helm Tiller Kubernetes error:__ The only error I am able to find in the logs is this. ``` tail -f /var/log/gitlab/gitlab-rails/kubernetes.log | jq ``` ```javascript { "severity": "INFO", "time": "2019-06-18T22:20:53.454Z", "correlation_id": "ERgbpB1VQt8", "service": "Clusters::Applications::InstallService", "app_id": 8, "app_name": "helm", "project_ids": [ 15 ], "group_ids": [], "event": "begin_install" } { "severity": "ERROR", "time": "2019-06-18T22:20:54.232Z", "correlation_id": "ERgbpB1VQt8", "exception": "Kubeclient::HttpError", "status_code": null, "namespace": "gitlab-managed-apps", "class_name": "Gitlab::Kubernetes::Namespace", "event": "failed_to_create_namespace", "message": "SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate)" } { "severity": "ERROR", "time": "2019-06-18T22:20:54.233Z", "correlation_id": "ERgbpB1VQt8", "error_code": null, "service": "Clusters::Applications::InstallService", "app_id": 8, "app_name": "helm", "project_ids": [ 15 ], "group_ids": [], "exception": "Kubeclient::HttpError", "message": "SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate)", "backtrace": [ "lib/gitlab/kubernetes/kube_client.rb:34:in `get_namespace'", "lib/gitlab/kubernetes/namespace.rb:14:in `exists?'", "lib/gitlab/kubernetes/namespace.rb:27:in `ensure_exists!'", "lib/gitlab/kubernetes/helm/api.rb:13:in `install'", "app/services/clusters/applications/install_service.rb:18:in `install'", "app/services/clusters/applications/install_service.rb:11:in `execute'", "app/workers/cluster_install_app_worker.rb:10:in `block in perform'", "app/workers/concerns/cluster_applications.rb:8:in `find_application'", "app/workers/cluster_install_app_worker.rb:9:in `perform'", "lib/gitlab/sidekiq_status/server_middleware.rb:7:in `call'", "lib/gitlab/sidekiq_middleware/correlation_logger.rb:10:in `block in call'", "lib/gitlab/sidekiq_middleware/correlation_logger.rb:9:in `call'", "lib/gitlab/sidekiq_middleware/batch_loader.rb:7:in `call'", "lib/gitlab/sidekiq_middleware/request_store_middleware.rb:8:in `call'", "lib/gitlab/sidekiq_middleware/memory_killer.rb:18:in `call'" ] } ``` I should be able to test if this is reproducible by creating two new VMs and installing everything again. I have not installed helm manually in GitLab because snapd will not run in the LXC without kernel changes to the Proxmox host. I have also not installed the helm binary in the $PATH as I don't know what the configuration would be. I want to be able to point different projects at their own Kubernetes cluster, so setting up helm manually would there to be one configuration for all projects in ~git. I really want to avoid that. Looking for advice from the GitLab experts. On a side note, the __API URL__ ignores the port __:16443__ so on the Kubernetes VM I set a firewall rule to redirect the connection. ``` iptables -t nat -A PREROUTING -p tcp --dport 443 -j REDIRECT --to-ports 16443 ``` I want to avoid this and get GitLab using 16443 for API calls as I would like the container to use 443.