426 Upgrade Required error

I installed a k8s cluster and the gitlab agent and registered it at another self-hosted gitlab instance. The kube context is properly injected into the CI job container:

$ kubectl config get-contexts
CURRENT   NAME                                                 CLUSTER   AUTHINFO   NAMESPACE
          varac-projects/kubernetes-agent-setup:varac-agent1   gitlab    agent:4    
++ echo '$ kubectl config use-context varac-projects/kubernetes-agent-setup:varac-agent1'
++ kubectl config use-context varac-projects/kubernetes-agent-setup:varac-agent1
$ kubectl config use-context varac-projects/kubernetes-agent-setup:varac-agent1
Switched to context "varac-projects/kubernetes-agent-setup:varac-agent1".

But using kubectl fails:

++ echo '$ kubectl version'
++ kubectl version
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.8", GitCommit:"7061dbbf75f9f82e8ab21f9be7e8ffcaae8e0d44", GitTreeState:"clean", BuildDate:"2022-03-16T14:10:06Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}
Error from server: the server responded with the status code 426 but did not return more information

Increasing the kubectl verbosity shows:

++ kubectl -v 10 version
I0324 14:36:01.894108      25 loader.go:372] Config loaded from file:  /builds/varac-projects/testproject.tmp/KUBECONFIG
I0324 14:36:01.894658      25 round_trippers.go:435] curl -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.22.8 (linux/amd64) kubernetes/7061dbb" -H "Authorization: Bearer <masked>" 'https://0xacab.org/-/kubernetes-agent/k8s-proxy/version?timeout=32s'
I0324 14:36:02.527251      25 round_trippers.go:454] GET https://0xacab.org/-/kubernetes-agent/k8s-proxy/version?timeout=32s 426 Upgrade Required in 632 milliseconds
I0324 14:36:02.527274      25 round_trippers.go:460] Response Headers:
I0324 14:36:02.527282      25 round_trippers.go:463]     X-Content-Type-Options: nosniff
I0324 14:36:02.527286      25 round_trippers.go:463]     Server: nginx
I0324 14:36:02.527290      25 round_trippers.go:463]     Date: Thu, 24 Mar 2022 14:36:02 GMT
I0324 14:36:02.527446      25 round_trippers.go:463]     Content-Type: text/plain; charset=utf-8
I0324 14:36:02.527456      25 round_trippers.go:463]     Content-Length: 81
I0324 14:36:02.527525      25 round_trippers.go:463]     Upgrade: websocket
I0324 14:36:02.531758      25 request.go:1181] Response Body: WebSocket protocol violation: Connection header "close" does not contain Upgrade
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.8", GitCommit:"7061dbbf75f9f82e8ab21f9be7e8ffcaae8e0d44", GitTreeState:"clean", BuildDate:"2022-03-16T14:10:06Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}
I0324 14:36:02.534095      25 helpers.go:217] server response object: [{
  "metadata": {},
  "status": "Failure",
  "message": "the server responded with the status code 426 but did not return more information",
  "details": {
    "causes": [
      {
        "reason": "UnexpectedServerResponse",
        "message": "WebSocket protocol violation: Connection header \"close\" does not contain Upgrade"
      }
    ]
  },
  "code": 426
}]
F0324 14:36:02.534117      25 helpers.go:116] Error from server: the server responded with the status code 426 but did not return more information
...

What's possibly going wrong ? The self-hosted gitlab site uses a bring-your-own nginx proxy in front of gitlab, and the config can be seen in this issue. Is there a nginx mis-configuration, or is this a bug in KAS ?

Possibly related issue I found:

• gitlab-org/gitlab#351477 (closed)
• #198 (closed)