Gitlab Kubernetes Agent and kubectl v1.27.0

Summary

2023-04-11 kubernetes v1.27.0 was released (https://kubernetes.io/releases/). So did kubectl.

After upgrade of kubectl in Gitlab CI job - it no longer works.

Steps to reproduce

  1. Use Gitlab Kubernetes Agent
  2. Run CI Job
deploy:
  image: alpine:latest
  script:
    - apk add --no-cache curl
    - curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.27.0/bin/linux/amd64/kubectl
    - chmod +x ./kubectl
    - mv ./kubectl /usr/local/bin/
    - kubectl config use-context ${KUBE_SELECTED_CONTEXT}
    - kubectl apply -f some-deployment.yml -v=8

What is the current bug behavior?

Error: responded with the status code 426

Requests logs on v1.27.0:

I0413 06:50:46.877519      67 loader.go:373] Config loaded from file:  /builds/kazinsys-dev/internal/task-hub.tmp/KUBECONFIG
I0413 06:50:46.878244      67 round_trippers.go:463] GET https://kas.gitlab.com/k8s-proxy/openapi/v2?timeout=32s
I0413 06:50:46.878271      67 round_trippers.go:469] Request Headers:
I0413 06:50:46.878291      67 round_trippers.go:473]     Authorization: Bearer <masked>
I0413 06:50:46.878306      67 round_trippers.go:473]     Accept: application/com.github.proto-openapi.spec.v2@v1.0+protobuf
I0413 06:50:46.878321      67 round_trippers.go:473]     User-Agent: kubectl/v1.27.0 (linux/amd64) kubernetes/1b4df30
I0413 06:50:47.793655      67 round_trippers.go:574] Response Status: 200 OK in 915 milliseconds
I0413 06:50:47.793693      67 round_trippers.go:577] Response Headers:
I0413 06:50:47.793706      67 round_trippers.go:580]     Cache-Control: no-cache, private
I0413 06:50:47.793716      67 round_trippers.go:580]     Content-Type: application/octet-stream
I0413 06:50:47.793726      67 round_trippers.go:580]     X-Kubernetes-Pf-Flowschema-Uid: caef336d-6d35-4082-b32c-5e31d0f9d35c
I0413 06:50:47.793737      67 round_trippers.go:580]     Date: Thu, 13 Apr 2023 06:50:47 GMT
I0413 06:50:47.793747      67 round_trippers.go:580]     Etag: <masked>
I0413 06:50:47.793758      67 round_trippers.go:580]     Last-Modified: Fri, 03 Mar 2023 18:30:10 GMT
I0413 06:50:47.793767      67 round_trippers.go:580]     X-Kubernetes-Pf-Prioritylevel-Uid: <masked>
I0413 06:50:47.793777      67 round_trippers.go:580]     X-Varied-Accept: application/com.github.proto-openapi.spec.v2@v1.0+protobuf
I0413 06:50:47.793787      67 round_trippers.go:580]     Accept-Ranges: bytes
I0413 06:50:47.793797      67 round_trippers.go:580]     Audit-Id: <masked>
I0413 06:50:47.793807      67 round_trippers.go:580]     Vary: Accept-Encoding
I0413 06:50:47.793816      67 round_trippers.go:580]     Vary: Accept
I0413 06:50:47.793829      67 round_trippers.go:580]     Vary: Accept-Encoding
I0413 06:50:47.793839      67 round_trippers.go:580]     Vary: Accept
I0413 06:50:47.793848      67 round_trippers.go:580]     Via: 2.0 gitlab-agent/v15.11.0-rc2/9587898b
I0413 06:50:47.793858      67 round_trippers.go:580]     Via: gRPC/1.0 gitlab-kas/v15.10.0/v15.10.0
...
I0413 06:50:48.703833      67 round_trippers.go:463] GET https://kas.gitlab.com/k8s-proxy/openapi/v3?timeout=32s
I0413 06:50:48.703911      67 round_trippers.go:469] Request Headers:
I0413 06:50:48.703942      67 round_trippers.go:473]     User-Agent: kubectl/v1.27.0 (linux/amd64) kubernetes/1b4df30
I0413 06:50:48.703971      67 round_trippers.go:473]     Authorization: Bearer <masked>
I0413 06:50:48.704006      67 round_trippers.go:473]     Accept: application/json, */*
I0413 06:50:49.210396      67 round_trippers.go:574] Response Status: 200 OK in 506 milliseconds
I0413 06:50:49.210489      67 round_trippers.go:577] Response Headers:
I0413 06:50:49.210519      67 round_trippers.go:580]     Accept-Ranges: bytes
I0413 06:50:49.210546      67 round_trippers.go:580]     Content-Length: 10861
I0413 06:50:49.210575      67 round_trippers.go:580]     Date: Thu, 13 Apr 2023 06:50:49 GMT
I0413 06:50:49.210600      67 round_trippers.go:580]     Last-Modified: Thu, 13 Apr 2023 06:50:49 GMT
I0413 06:50:49.210633      67 round_trippers.go:580]     Audit-Id: <masked>
I0413 06:50:49.210659      67 round_trippers.go:580]     Cache-Control: no-cache, private
I0413 06:50:49.210684      67 round_trippers.go:580]     Content-Type: text/plain; charset=utf-8
I0413 06:50:49.210710      67 round_trippers.go:580]     Via: 2.0 gitlab-agent/v15.11.0-rc2/9587898b
I0413 06:50:49.210742      67 round_trippers.go:580]     Via: gRPC/1.0 gitlab-kas/v15.10.0/v15.10.0
I0413 06:50:49.210784      67 round_trippers.go:580]     X-Kubernetes-Pf-Flowschema-Uid: <masked>
I0413 06:50:49.210814      67 round_trippers.go:580]     X-Kubernetes-Pf-Prioritylevel-Uid: <masked>
...
I0413 06:50:49.218345      67 round_trippers.go:463] GET https://kas.gitlab.com/openapi/v3/apis/apps/v1?hash=<masked>&timeout=32s
I0413 06:50:49.218379      67 round_trippers.go:469] Request Headers:
I0413 06:50:49.218464      67 round_trippers.go:473]     Accept: application/json
I0413 06:50:49.218480      67 round_trippers.go:473]     User-Agent: kubectl/v1.27.0 (linux/amd64) kubernetes/1b4df30
I0413 06:50:49.218562      67 round_trippers.go:473]     Authorization: Bearer <masked>
I0413 06:50:49.408905      67 round_trippers.go:574] Response Status: 426 Upgrade Required in 190 milliseconds
I0413 06:50:49.408935      67 round_trippers.go:577] Response Headers:
I0413 06:50:49.408949      67 round_trippers.go:580]     Date: Thu, 13 Apr 2023 06:50:49 GMT
I0413 06:50:49.408959      67 round_trippers.go:580]     Content-Length: 76
I0413 06:50:49.408969      67 round_trippers.go:580]     Content-Type: text/plain; charset=utf-8
I0413 06:50:49.408991      67 round_trippers.go:580]     Server: gitlab-kas/v15.10.0/v15.10.0
I0413 06:50:49.409002      67 round_trippers.go:580]     X-Content-Type-Options: nosniff
I0413 06:50:49.409290      67 request.go:1188] Response Body: WebSocket protocol violation: Connection header "" does not contain Upgrade
error: error validating "deployment.yml": error validating data: the server responded with the status code 426 but did not return more information; if you choose to ignore these errors, turn validation off with --validate=false

Last Request (https://kas.gitlab.com/openapi/v3/apis/apps/v1) is made to an invalid URL.

What is the expected correct behavior?

Requests log in v1.26.3:

GET https://kas.gitlab.com/k8s-proxy/openapi/v2?timeout=32s
GET https://kas.gitlab.com/k8s-proxy/api?timeout=32s
GET https://kas.gitlab.com/k8s-proxy/apis?timeout=32s
GET https://kas.gitlab.com/k8s-proxy/apis/metrics.k8s.io/v1beta1?timeout=32s
GET https://kas.gitlab.com/k8s-proxy/apis/events.k8s.io/v1?timeout=32s
GET https://kas.gitlab.com/k8s-proxy/apis/authentication.k8s.io/v1?timeout=32s
GET https://kas.gitlab.com/k8s-proxy/apis/authorization.k8s.io/v1?timeout=32s
GET https://kas.gitlab.com/k8s-proxy/api/v1?timeout=32s
GET https://kas.gitlab.com/k8s-proxy/apis/autoscaling/v2?timeout=32s
GET https://kas.gitlab.com/k8s-proxy/apis/apiregistration.k8s.io/v1?timeout=32s
GET https://kas.gitlab.com/k8s-proxy/apis/apps/v1?timeout=32s
GET https://kas.gitlab.com/k8s-proxy/apis/autoscaling/v1?timeout=32s
...
<and many other successfull requests>

Environment

  • gitlab.com
  • Bare-metal K8S Cluster v1.25.6
  • Gitlab Kubernetes Agent v15.11.0-rc2
  • Gitlab Runner Kubernetes Executor v15.10.1

Related links

#351477 (closed)

https://forum.gitlab.com/t/gitlab-kas-kube-cluster-connection-woes/64197