Investigate proxy error: transport: the stream is done or WriteHeader was already called
$ kubectl get pods
Error from server (InternalError): an error on the server ("Proxy failed to read response from agent: rpc error: code = Internal desc = transport: transport: the stream is done or WriteHeader was already called") has prevented the request from succeeding
From https://gitlab.com/ash2k/agents-config/-/jobs/1579711021:
$ kubectl -v 10 version
I0910 10:47:19.167416 18 loader.go:372] Config loaded from file: /builds/ash2k/agents-config.tmp/KUBECONFIG
I0910 10:47:19.168427 18 round_trippers.go:435] curl -v -XGET -H "Authorization: Bearer <masked>" -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.22.1 (linux/amd64) kubernetes/632ed30" 'https://kas.gitlab.com/k8s-proxy/version?timeout=32s'
I0910 10:47:19.398004 18 round_trippers.go:454] GET https://kas.gitlab.com/k8s-proxy/version?timeout=32s 502 Bad Gateway in 229 milliseconds
I0910 10:47:19.398384 18 round_trippers.go:460] Response Headers:
I0910 10:47:19.398572 18 round_trippers.go:463] Content-Length: 100
I0910 10:47:19.398704 18 round_trippers.go:463] Via: 1.1 google
I0910 10:47:19.398899 18 round_trippers.go:463] Alt-Svc: clear
I0910 10:47:19.399083 18 round_trippers.go:463] Content-Type: text/plain; charset=utf-8
I0910 10:47:19.399258 18 round_trippers.go:463] Server: gitlab-kas/v14.2.2/2b22fa6
I0910 10:47:19.399433 18 round_trippers.go:463] X-Content-Type-Options: nosniff
I0910 10:47:19.399605 18 round_trippers.go:463] X-Request-Id: 01FF7N3802GSEF0K9MC8K24QVN
I0910 10:47:19.399793 18 round_trippers.go:463] Date: Fri, 10 Sep 2021 10:47:18 GMT
I0910 10:47:19.400107 18 request.go:1181] Response Body: Proxy failed to read response from agent: rpc error: code = Unavailable desc = transport is closing
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.1", GitCommit:"632ed300f2c34f6d6d15ca4cef3d3c7073412212", GitTreeState:"clean", BuildDate:"2021-08-19T15:45:37Z", GoVersion:"go1.16.7", Compiler:"gc", Platform:"linux/amd64"}
I0910 10:47:19.400817 18 helpers.go:217] server response object: [{
"metadata": {},
"status": "Failure",
"message": "an error on the server (\"Proxy failed to read response from agent: rpc error: code = Unavailable desc = transport is closing\") has prevented the request from succeeding",
"reason": "InternalError",
"details": {
"causes": [
{
"reason": "UnexpectedServerResponse",
"message": "Proxy failed to read response from agent: rpc error: code = Unavailable desc = transport is closing"
}
]
},
"code": 502
}]
F0910 10:47:19.401237 18 helpers.go:116] Error from server (InternalError): an error on the server ("Proxy failed to read response from agent: rpc error: code = Unavailable desc = transport is closing") has prevented the request from succeeding
Logs:
{
"_index": "pubsub-kas-inf-gprd-000033",
"_type": "_doc",
"_id": "dXOpy3sB-RTw6rRRBJNr",
"_version": 1,
"_score": null,
"_source": {
"@timestamp": "2021-09-09T17:44:15.588Z",
"kubernetes": {
"container_image": "dev.gitlab.org:5005/gitlab/gitlab-agent/kas:v14.2.2",
"host": "gke-gprd-gitlab-gke-default-2-216a4699-fd06",
"container_name": "kas",
"namespace_name": "gitlab",
"pod_name": "gitlab-kas-6985dfbdb9-rkvlw",
"region": "us-east1"
},
"json": {
"mod_name": "kubernetes_api",
"correlation_id": "01FF5TG4DHJS2XQSX9EAV7PCSV",
"shard": "default",
"agent_id": 930,
"error": "rpc error: code = Internal desc = transport: transport: the stream is done or WriteHeader was already called",
"stage": "main",
"tier": "sv",
"tag": "gitlab-kas.var.log.containers.gitlab-kas-6985dfbdb9-rkvlw_gitlab_kas-ee87fd5185dacc39296b5a4100b4179dd9af0ece2d75e917e0dc7742f74ed3bd.log",
"level": "error",
"environment": "gprd",
"msg": "Proxy failed to read response from agent",
"type": "kas",
"time": "2021-09-09T17:43:14.613Z"
},
"type": "pubsubbeat-pubsub-kas-inf-gprd-77f997bd8c-8qmnl",
"publish_time": "2021-09-09T17:44:15.566Z",
"host": {
"name": "pubsubbeat-pubsub-kas-inf-gprd-77f997bd8c-8qmnl"
}
},
"fields": {
"publish_time": [
"2021-09-09T17:44:15.566Z"
],
"@timestamp": [
"2021-09-09T17:44:15.588Z"
],
"json.time": [
"2021-09-09T17:43:14.613Z"
]
},
"sort": [
1631209394613
]
}
Things to investigate:
-
Request router in kas -
kas -> kas gRPC connectivity (keep-alives?) issues
Edited by Mikhail Mazurskiy