Add appropriate infrastructure support for KAS to leverage SPDY
Context
kubect exec/attach/cp
calls do not work on gitlab.com with the agent for Kubernetes using the CI/CD workflow. We found that these commands fail for two reasons:
-
kubectl
and Kube API server does not support Websockets - the load balancer used on gitlab.com does not support SPDY
This is the most often reported issue for the agent, an issue that limits many of them to migrate from cert-based cluster connections to agent-based connections. As a result, it's a top priority to fix for ~"group::configure".
While we plan to improve the Kube API server, the earliest release of the fix would happen in Kubernetes 1.26. As a result, we are looking for alternative solutions.
We think that we could get kubectl exec
support on gitlab.com by changing the load balancer.
After a lot of work in preprod, we've settled on performing the following:
- Move KAS behind HAProxy
Milestones
-
KAS features work on preprod -
KAS features work in staging - gitlab-com/gl-infra/delivery#2577 (closed) -
Migration planning, and CR work identified for production - gitlab-com/gl-infra/production#7855 (closed) -
KAS features work in production
Scope
Now that self-managed users are no longer affected by the exec/attach/cp/port-forward issue, we have only GitLab.com to worry about. What if we switch the load balancer into TCP mode for GitLab.com only? This issue tracks the work to investigate that.
Known work items for such change or things to investigate:
-
Can GKE load balancer work in TLS+TCP mode i.e. terminate TLS and pass cleartext connection to the packend? Nope -
If the above is not possible, how can we configure kas with TLS certificates? kas supports that. Check chart and find out how certs can be provisioned in the GitLab.com deployment. Nope -
Move KAS behind HAProxy - This is the chosen solution