gitlab-kas not recognizing custom certificates in user specified directory
Issue
When using custom self signed certificates, the agent within an external cluster was unable to connect to the gRPC endpoint.
Self-hosted Gitlab
gitlab-ctl tail gitlab-kas
{"level":"error","time":"2022-12-14T20:20:32.658Z","msg":"AgentInfo()","grpc_service":"gitlab.agent.reverse_tunnel.rpc.ReverseTunnel","grpc_method":"Connect","error":"Get \"https://gitlab.xxx.domain/api/v4/internal/kubernetes/agent_info\": x509: certificate signed by unknown authority"}
External K8s
Deploying the agent.
helm repo add gitlab https://charts.gitlab.io
helm repo update
helm upgrade --install agent gitlab/gitlab-agent \
--namespace gitlab-agent \
--create-namespace \
--set image.tag=v15.4.0 \
--set config.token=XXX...XXX \
--set config.kasAddress=wss://gitlab.xxx.domain:443/-/kubernetes-agent/ \
--set config.caCert="$(cat gitlab.xxx.domain.crt)"
kubectl -n gitlab-agent logs pod/agent-gitlab-agent-xxx-xxx
{"level":"info","time":"2022-12-15T13:49:39.671Z","msg":"Observability endpoint is up","mod_name":"observability","net_network":"tcp","net_address":"[::]:8080"}
Setting the gitlab-kas environment variable SSL_CERT_DIR
within /etc/gitlab/gitlab.rb and running gitlab-ctl reconfigure
did not work even though gitlab.xxx.domain.crt was within the directory.
gitlab_kas['env'] = {
'SSL_CERT_DIR => '/etc/gitlab/ssl'
}
Resolution
Setting SSL_CERT_DIR
to /opt/gitlab/embedded/ssl/certs
and moving gitlab.xxx.domain.crt to /etc/gitlab/trusted_certs
allows the gRPC endpoint to recognize the certificates after gitlab-ctl reconfigure
.
Recommendation
Documenting that custom certificates must be placed in /etc/gitlab/trusted_certs
within the Troubleshooting the GitLab Agent for Kubernetes section.