gitlab-kas not recognizing custom certificates in user specified directory

Issue

When using custom self signed certificates, the agent within an external cluster was unable to connect to the gRPC endpoint.

Self-hosted Gitlab

gitlab-ctl tail gitlab-kas

{"level":"error","time":"2022-12-14T20:20:32.658Z","msg":"AgentInfo()","grpc_service":"gitlab.agent.reverse_tunnel.rpc.ReverseTunnel","grpc_method":"Connect","error":"Get \"https://gitlab.xxx.domain/api/v4/internal/kubernetes/agent_info\": x509: certificate signed by unknown authority"}

External K8s

Deploying the agent.

helm repo add gitlab https://charts.gitlab.io
helm repo update
helm upgrade --install agent gitlab/gitlab-agent \
    --namespace gitlab-agent \
    --create-namespace \
    --set image.tag=v15.4.0 \
    --set config.token=XXX...XXX \
    --set config.kasAddress=wss://gitlab.xxx.domain:443/-/kubernetes-agent/ \
--set config.caCert="$(cat gitlab.xxx.domain.crt)"

kubectl -n gitlab-agent logs pod/agent-gitlab-agent-xxx-xxx

{"level":"info","time":"2022-12-15T13:49:39.671Z","msg":"Observability endpoint is up","mod_name":"observability","net_network":"tcp","net_address":"[::]:8080"}

Setting the gitlab-kas environment variable SSL_CERT_DIR within /etc/gitlab/gitlab.rb and running gitlab-ctl reconfigure did not work even though gitlab.xxx.domain.crt was within the directory.

gitlab_kas['env'] = {
    'SSL_CERT_DIR => '/etc/gitlab/ssl'
}

Resolution

Setting SSL_CERT_DIR to /opt/gitlab/embedded/ssl/certs and moving gitlab.xxx.domain.crt to /etc/gitlab/trusted_certs allows the gRPC endpoint to recognize the certificates after gitlab-ctl reconfigure.

Recommendation

Documenting that custom certificates must be placed in /etc/gitlab/trusted_certs within the Troubleshooting the GitLab Agent for Kubernetes section.