Skip to content

KAS does not work with GitLab on relative URL

Trying to get KAS working with GitLab deployed on a relative URL, e.g. http://example.org/gitlab, I cannot seem to get the agent to connect with the server. I deployed my agent with a kasAddress of ws://example.org/gitlab/-/kubernetes-agent/ (see also: gitlab-org/omnibus-gitlab#6855 (closed)) but the web UI stubbornly keeps showing that the agent has never connected.

I'm running the gitlab/gitlab-ce:15.0.0-ce.0 image with

external_url "http://example.org/gitlab"
gitlab_rails['gitlab_kas_external_url'] = "http://example.org/gitlab"
gitlab_rails['gitlab_kas_external_k8s_proxy_url'] = "http://example.org/gitlab/-/kubernetes-agent/"

I added the last two in case it mattered but that does not appear to be the case.
IMHO, they should not be needed if external_url provides the right "prefix".

The symptoms I see on the agent side are the same as reported in #252 (closed).

{"level":"info","time":"2022-05-27T05:52:20.817Z","msg":"Observability endpoint is up","mod_name":"observability","net_network":"tcp","net_address":"[::]:8080"}
{"level":"error","time":"2022-05-27T05:52:20.821Z","msg":"Error handling a connection","mod_name":"reverse_tunnel","error":"rpc error: code = NotFound desc = agent not found","correlation_id":"01G4215AANRQVAGNPJ1DE2R783"}
{"level":"error","time":"2022-05-27T05:52:20.822Z","msg":"Error handling a connection","mod_name":"reverse_tunnel","error":"rpc error: code = NotFound desc = agent not found","correlation_id":"01G4215AANRQVAGNPJ1DGVME59"}
{"level":"warn","time":"2022-05-27T05:52:20.822Z","msg":"GetConfiguration.Recv failed","error":"rpc error: code = NotFound desc = agent not found","correlation_id":"01G4215AANRQVAGNPJ1A1WMRN5"}

On the GitLab instance side, /var/log/gitlab/gitlab-kas/current shows

2022-05-27_05:49:20.54202 {"level":"info","time":"2022-05-27T05:49:20.541Z","msg":"Observability endpoint is up","mod_name":"observability","net_network":"tcp","net_address":"127.0.0.1:8151"}
2022-05-27_05:49:20.54210 {"level":"info","time":"2022-05-27T05:49:20.541Z","msg":"Private API endpoint is up","net_network":"tcp","net_address":"127.0.0.1:8155"}
2022-05-27_05:49:20.54211 {"level":"info","time":"2022-05-27T05:49:20.542Z","msg":"Agentk API endpoint is up","net_network":"tcp","net_address":"127.0.0.1:8150","is_websocket":true}
2022-05-27_05:49:20.54211 {"level":"info","time":"2022-05-27T05:49:20.542Z","msg":"API endpoint is up","net_network":"tcp","net_address":"127.0.0.1:8153"}
2022-05-27_05:49:20.54216 {"level":"info","time":"2022-05-27T05:49:20.542Z","msg":"Kubernetes API endpoint is up","mod_name":"kubernetes_api","net_network":"tcp","net_address":"127.0.0.1:8154"}

and nothing changes after that.

In /var/log/gitlab/nginx/gitlab_access.log, however, I get

10.11.55.228 - - [27/May/2022:05:52:18 +0000] "GET /gitlab/-/kubernetes-agent/ HTTP/1.1" 101 414 "" "gitlab-agent/v15.0.0/b93d7a6" -
172.20.0.1 - - [27/May/2022:05:53:00 +0000] "GET /api/v4/internal/kubernetes/agent_info HTTP/1.1" 404 1625 "" "gitlab-kas/v15.0.0/b93d7a6" 1.99
172.20.0.1 - - [27/May/2022:05:54:04 +0000] "GET /api/v4/internal/kubernetes/agent_info HTTP/1.1" 404 1625 "" "gitlab-kas/v15.0.0/b93d7a6" 1.99

and, correspondingly, in /var/log/gitlab/nginx/gitlab_error.log

2022/05/27 05:53:00 [error] 486#0: *16 open() "/var/opt/gitlab/nginx/html/api/v4/internal/kubernetes/agent_info" failed (2: No such file or directory), client: 172.20.0.1, server: maniac.machine.easy, request: "GET /api/v4/internal/kubernetes/agent_info HTTP/1.1", host: "maniac.machine.easy"
2022/05/27 05:54:04 [error] 486#0: *19 open() "/var/opt/gitlab/nginx/html/api/v4/internal/kubernetes/agent_info" failed (2: No such file or directory), client: 172.20.0.1, server: maniac.machine.easy, request: "GET /api/v4/internal/kubernetes/agent_info HTTP/1.1", host: "maniac.machine.easy"

The Nginx messages repeat every time agentk tries to receive a response to GetConfiguration (except for the GET /gitlab/-/kubernetes-agent/ request).

I strongly suspect that agentk does not support GitLab on a relative URL, yet. If that's indeed the case, a fix would be most appreciated 🙇