GitLab needs to be able to talk to GitLab Kubernetes Agent Server (`kas`) to: - get information about connected agents - interact with agents - interact with Kubernetes clusters Agents connect to `kas` and keep an open connection. When GitLab needs to talk to a particular agent, a `kas` instance, that this agent is connected to, needs to be found and the request needs to be routed to it. This epic is about: - finding and documenting a way to solve this. https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/merge_requests/90 - implementing it. # Implementation / iterations ## Iteration 1: tracking stuff in Redis 1. [x] Configuration and any needed plumbing to connect to Redis. Was done as part of https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/merge_requests/103. 1. [x] https://gitlab.com/gitlab-org/gitlab/-/issues/277316 Use environment variables populated by [downward API](https://kubernetes.io/docs/tasks/inject-data-application/environment-variable-expose-pod-information/#the-downward-api) to supply `Pod` namespace and name to `agentk` container. 1. [x] https://gitlab.com/gitlab-org/gitlab/-/issues/277317 `agentk` should send information about itself in `GetConfiguration` request. 1. [x] Track connected agents in Redis. https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/merge_requests/203 ### Iteration 2: beginning of external API 1. [x] Implement `GetConnectedAgentsForProjectId()`. https://gitlab.com/gitlab-org/gitlab/-/issues/293969 1. [x] Implement `GetConnectedAgentsForAgentId()`. https://gitlab.com/gitlab-org/gitlab/-/issues/293970 ### Iteration 3: `agentk` -> `kas` reverse tunnel 1. [x] Establish "tunnel" connection from agent to kas. https://gitlab.com/gitlab-org/gitlab/-/issues/296824 1. [x] Track connected "tunnels" from agents in Redis. https://gitlab.com/gitlab-org/gitlab/-/issues/293890 ### Iteration 4: `kas` -> `kas` communication 1. [x] Implement proper request routing using the tunnels, tracked in Redis. https://gitlab.com/gitlab-org/gitlab/-/issues/297658 1. [ ] Publish pub-sub events when tunnel connects. https://gitlab.com/gitlab-org/gitlab/-/issues/320732 1. [ ] Consume pub-sub events to reduce tunnel lookup latency. https://gitlab.com/gitlab-org/gitlab/-/issues/323131 ### Features for customers (not part of this epic) - https://gitlab.com/gitlab-org/gitlab/-/issues/240918 - https://gitlab.com/gitlab-org/gitlab/-/issues/280563 - ...