Skip to content

Remove Kubernetes proxy write timeout

Mikhail Mazurskiy requested to merge ash2k/bump-timeouts into master

Fixes #255 (closed).

Fixes https://gitlab.com/gitlab-com/gl-infra/gitlab-dedicated/team/-/issues/671.

I have managed to reproduce the issue, although I'm getting a different error. This is probably because I don't have any load balancer in front of kas' Kubernetes proxy endpoint.

Setup:

  • GDK
  • agentk to kas via direct grpc (no nginx, no websockets)
  • kubectl -> kas directly (no nginx)
  • rancher desktop v1.5.1, Kubernetes v1.24.3

Without the fix:

date && kubectl logs  --all-containers -f -n kube-system coredns-d76bd69b-g84qz -v 9 || date
Thu 25 Aug 2022 22:38:02 AEST
I0825 22:38:02.757075   52392 loader.go:374] Config loaded from file:  /Users/mike/.kube/config
I0825 22:38:02.759703   52392 round_trippers.go:466] curl -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.25.0 (darwin/arm64) kubernetes/a866cbe" 'http://gdk.test:8154/-/k8s-proxy/api/v1/namespaces/kube-system/pods/coredns-d76bd69b-g84qz'
I0825 22:38:02.761155   52392 round_trippers.go:495] HTTP Trace: DNS Lookup for gdk.test resolved to [{172.16.123.1 }]
I0825 22:38:02.761422   52392 round_trippers.go:510] HTTP Trace: Dial to tcp:172.16.123.1:8154 succeed
I0825 22:38:02.783655   52392 round_trippers.go:553] GET http://gdk.test:8154/-/k8s-proxy/api/v1/namespaces/kube-system/pods/coredns-d76bd69b-g84qz 200 OK in 23 milliseconds
I0825 22:38:02.783670   52392 round_trippers.go:570] HTTP Statistics: DNSLookup 1 ms Dial 0 ms TLSHandshake 0 ms ServerProcessing 22 ms Duration 23 ms
I0825 22:38:02.783675   52392 round_trippers.go:577] Response Headers:
I0825 22:38:02.783679   52392 round_trippers.go:580]     X-Request-Id: 01GBAG4VR9RMG78K32H3S6EXAQ
I0825 22:38:02.783683   52392 round_trippers.go:580]     Date: Thu, 25 Aug 2022 12:38:02 GMT
I0825 22:38:02.783687   52392 round_trippers.go:580]     Cache-Control: no-cache, private
I0825 22:38:02.783690   52392 round_trippers.go:580]     Content-Type: application/json
I0825 22:38:02.783693   52392 round_trippers.go:580]     Via: 1.1 gitlab-agent/v0.0.0/00000000
I0825 22:38:02.783697   52392 round_trippers.go:580]     Via: gRPC/1.0 gitlab-kas/v0.0.0/00000000
I0825 22:38:02.783700   52392 round_trippers.go:580]     X-Kubernetes-Pf-Flowschema-Uid: 0848073f-9750-4291-925c-57643e024a51
I0825 22:38:02.783704   52392 round_trippers.go:580]     X-Kubernetes-Pf-Prioritylevel-Uid: 549b83f7-181d-45c6-90fe-f6df5c20e110
I0825 22:38:02.783707   52392 round_trippers.go:580]     Audit-Id: 1429e569-f671-4e73-abf2-b270e52fcea9
I0825 22:38:02.785462   52392 request.go:1073] Response Body: ...
I0825 22:38:02.786834   52392 round_trippers.go:466] curl -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.25.0 (darwin/arm64) kubernetes/a866cbe" 'http://gdk.test:8154/-/k8s-proxy/api/v1/namespaces/kube-system/pods/coredns-d76bd69b-g84qz/log?container=coredns&follow=true'
I0825 22:38:02.810774   52392 round_trippers.go:553] GET http://gdk.test:8154/-/k8s-proxy/api/v1/namespaces/kube-system/pods/coredns-d76bd69b-g84qz/log?container=coredns&follow=true 200 OK in 23 milliseconds
I0825 22:38:02.810804   52392 round_trippers.go:570] HTTP Statistics: GetConnection 0 ms ServerProcessing 23 ms Duration 23 ms
I0825 22:38:02.810809   52392 round_trippers.go:577] Response Headers:
I0825 22:38:02.810814   52392 round_trippers.go:580]     Audit-Id: 08821e73-6146-4f91-ada8-d7d1b9c6bd49
I0825 22:38:02.810818   52392 round_trippers.go:580]     Cache-Control: no-cache, private
I0825 22:38:02.810822   52392 round_trippers.go:580]     X-Request-Id: 01GBAG4VS2T2S1PQQC45E18XKR
I0825 22:38:02.810825   52392 round_trippers.go:580]     Content-Type: text/plain
I0825 22:38:02.810829   52392 round_trippers.go:580]     Date: Thu, 25 Aug 2022 12:38:02 GMT
I0825 22:38:02.810832   52392 round_trippers.go:580]     Via: 1.1 gitlab-agent/v0.0.0/00000000
I0825 22:38:02.810836   52392 round_trippers.go:580]     Via: gRPC/1.0 gitlab-kas/v0.0.0/00000000
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
.:53
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[INFO] plugin/reload: Running configuration SHA512 = b941b080e5322f6519009bb49349462c7ddb6317425b0f6a83e5451175b720703949e3f3b454a24e77f3ffe57fd5e9c6130e528a5a1dd00d9000e4afd6c1108d
CoreDNS-1.9.1
linux/arm64, go1.17.8, 4b597f8
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
...
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
error: unexpected EOF
Thu 25 Aug 2022 22:38:30 AEST

With the fix - no error, log tailing is not getting interrupted so quickly. Note that log tailing is still limited by max connection age that kas sets. It is currently set to 30 minutes.

Edited by Mikhail Mazurskiy

Merge request reports