Gitlab agent on k8s: Failed to register agent pod. Please make sure the agent version matches the server version

Summary

I'm trying to install gitlab agent on a K3S cluster using Helm exactly the command provided from the gitlab kubernetes cluster dashboard:

Steps to reproduce

helm upgrade --install k8s-connection-poc gitlab/gitlab-agent
--namespace gitlab-agent-k8s-connection-poc
--create-namespace
--set image.tag=v16.8.0-rc1
--set config.token=token
--set config.kasAddress=wss://URL/-/kubernetes-agent/
-f gitlab-agent-values.yaml

Configuration used

in this values file "gitlab-agent-values.yaml" I added the secret as volume for the SSL self-signed certificate:

extraVolumes:
  - name: kas-cert-ca
    secret:
      secretName: kas-cert-ca
extraVolumeMounts:
  - name: kas-cert-ca
    mountPath: /etc/ssl/certs/ca-certificates.crt
    subPath: ca.crt

Current behavior

But the agent is not connecting

when I check the logs of the pods I get this: "msg":"Failed to register agent pod. Please make sure the agent version matches the server version","mod_name":"agent_registrar","error":"rpc error: code = Unavailable desc = unavailable"}

Versions

• Chart: gitlab-agent-2.0.0

• Platform:

  • Self-hosted: Rancher K3S

• Kubernetes: (kubectl version)

  • Client Version: v1.29.1
  • Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
  • Server Version: v1.28.4+k3s2

• Helm: (helm version)

  • version.BuildInfo{Version:"v3.14.1", GitCommit:"e8858f8696b144ee7c533bd9d49a353ee6c4b98d", GitTreeState:"clean", GoVersion:"go1.21.7"}

Relevant logs

{"level":"info","time":"2024-05-22T07:39:03.300Z","msg":"Observability endpoint is up","mod_name":"observability","net_network":"tcp","net_address":"[::]:8080"}
{"level":"error","time":"2024-05-22T07:39:03.393Z","msg":"Failed to register agent pod. Please make sure the agent version matches the server version","mod_name":"agent_registrar","error":"rpc error: code = Unavailable desc = unavailable"}
{"level":"error","time":"2024-05-22T07:39:17.595Z","msg":"Failed to register agent pod. Please make sure the agent version matches the server version","mod_name":"agent_registrar","error":"rpc error: code = Unavailable desc = unavailable"}