Gitlab k8s agent: KAS unable to upgrade connection

I have setup gitlab agent on my k8s cluster and integrated with CICD. I can use some commands like kubectl get pods, kubectl apply, kubectl delete, helm install/upgrade from CICD pipeline. However, I cannot use kubectl cp to copy some files to containers. Here is the debug log from the command

[0KRunning with gitlab-runner 14.7.0~beta.58.gfa48f33b (fa48f33b)[0;m
[0K  on green-4.shared.runners-manager.gitlab.com/default ntHFEtyX[0;m
section_start:1642737114:prepare_executor
[0K[0K[36;1mPreparing the "docker+machine" executor[0;m[0;m
[0KUsing Docker executor with image bitnami/kubectl:latest ...[0;m
[0KPulling docker image bitnami/kubectl:latest ...[0;m
[0KUsing docker image sha256:fb24b9c740f1e6374aeb3e91cec4a8ba96532ae7d4eea050d0eac06a5a4d3d8b for bitnami/kubectl:latest with digest bitnami/kubectl@sha256:fc783a51918d659bae80f9a0341c1e8b813a9ffba51b852f22844571cc36ac53 ...[0;m
section_end:1642737126:prepare_executor
[0Ksection_start:1642737126:prepare_script
[0K[0K[36;1mPreparing environment[0;m[0;m
Running on runner-nthfetyx-project-32984389-concurrent-0 via runner-nthfetyx-shared-1642737070-7ae2d438...
section_end:1642737128:prepare_script
[0Ksection_start:1642737128:get_sources
[0K[0K[36;1mGetting source from Git repository[0;m[0;m
[32;1m$ eval "$CI_PRE_CLONE_SCRIPT"[0;m
[32;1mFetching changes with git depth set to 20...[0;m
Initialized empty Git repository in /builds/the.tran/gitlab-agent-demo/.git/
[32;1mCreated fresh repository.[0;m
[32;1mChecking out 9e06ee7f as main...[0;m

[32;1mSkipping Git submodules setup[0;m
section_end:1642737129:get_sources
[0Ksection_start:1642737129:step_script
[0K[0K[36;1mExecuting "step_script" stage of the job script[0;m[0;m
[0KUsing docker image sha256:fb24b9c740f1e6374aeb3e91cec4a8ba96532ae7d4eea050d0eac06a5a4d3d8b for bitnami/kubectl:latest with digest bitnami/kubectl@sha256:fc783a51918d659bae80f9a0341c1e8b813a9ffba51b852f22844571cc36ac53 ...[0;m
[32;1m$ export DEPLOYMENT=ttran-dev[0;m
[32;1m$ kubectl config use-context the.tran/gitlab-agent-demo:paas-eu-demo[0;m
Switched to context "the.tran/gitlab-agent-demo:paas-eu-demo".
[32;1m$ kubectl -n ${DEPLOYMENT} get pods -l "app=nginx" -o name | sed 's/^pod\///' > pods.txt[0;m
[32;1m$ cat pods.txt[0;m
nginx-6fff489d7c-8llx6
[32;1m$ ls -l[0;m
total 20
-rw-rw-rw- 1 root root 7618 Jan 21 03:52 README.md
-rw-rw-rw- 1 root root  837 Jan 21 03:52 nginx-deployment.yaml
-rw-r--r-- 1 1001 root   23 Jan 21 03:52 pods.txt
drwxrwxrwx 2 root root 4096 Jan 21 03:52 sync
[32;1m$ for pod in `cat pods.txt`; do # collapsed multi-line command[0;m
I0121 03:52:13.383679      22 loader.go:372] Config loaded from file:  /builds/the.tran/gitlab-agent-demo.tmp/KUBECONFIG
I0121 03:52:13.385491      22 round_trippers.go:466] curl -v -XGET  -H "User-Agent: kubectl/v1.23.2 (linux/amd64) kubernetes/9d14243" -H "Accept: application/json, */*" -H "Authorization: Bearer <masked>" 'https://kas.gitlab.com/k8s-proxy/api/v1/namespaces/ttran-dev/pods/nginx-6fff489d7c-8llx6'
I0121 03:52:13.387725      22 round_trippers.go:495] HTTP Trace: DNS Lookup for kas.gitlab.com resolved to [{172.65.247.105 } {2606:4700:90:0:16ea:650c:2f4d:2c80 }]
I0121 03:52:13.398785      22 round_trippers.go:510] HTTP Trace: Dial to tcp:172.65.247.105:443 succeed
I0121 03:52:13.645137      22 round_trippers.go:570] HTTP Statistics: DNSLookup 1 ms Dial 10 ms TLSHandshake 36 ms ServerProcessing 209 ms Duration 259 ms
I0121 03:52:13.645535      22 round_trippers.go:577] Response Headers:
I0121 03:52:13.645819      22 round_trippers.go:580]     Content-Type: application/json
I0121 03:52:13.646147      22 round_trippers.go:580]     Date: Fri, 21 Jan 2022 03:52:12 GMT
I0121 03:52:13.646357      22 round_trippers.go:580]     X-Request-Id: 01FSXC6RX1216PS8W6YFEN1F05
I0121 03:52:13.646482      22 round_trippers.go:580]     X-Kubernetes-Pf-Prioritylevel-Uid: bde5d280-854a-4ee8-ae4b-453ac930146b
I0121 03:52:13.646666      22 round_trippers.go:580]     Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
I0121 03:52:13.646788      22 round_trippers.go:580]     Audit-Id: 6f667034-2ad2-41da-b75d-c02b0a813569
I0121 03:52:13.646992      22 round_trippers.go:580]     Cache-Control: no-cache, private
I0121 03:52:13.647134      22 round_trippers.go:580]     Via: 2.0 gitlab-agent/v14.5.0/523894a
I0121 03:52:13.647320      22 round_trippers.go:580]     Via: gRPC/1.0 gitlab-kas/v14.5.0/523894a, 1.1 google
I0121 03:52:13.647440      22 round_trippers.go:580]     X-Kubernetes-Pf-Flowschema-Uid: e797b352-8a5f-4baf-9291-df57629e57c6
I0121 03:52:13.647742      22 request.go:1181] Response Body: {"kind":"Pod","apiVersion":"v1","metadata":{"name":"nginx-6fff489d7c-8llx6","generateName":"nginx-6fff489d7c-","namespace":"ttran-dev","uid":"f4239bcb-a048-491a-b93e-d7deaee3155a","resourceVersion":"21878774","creationTimestamp":"2022-01-21T03:44:57Z","labels":{"app":"nginx","pod-template-hash":"6fff489d7c"},"annotations":{"kubernetes.io/psp":"eks.privileged"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"nginx-6fff489d7c","uid":"0fa06b5f-d1a9-46a4-8284-f4bd22aeacde","controller":true,"blockOwnerDeletion":true}],"managedFields":[{"manager":"kube-controller-manager","operation":"Update","apiVersion":"v1","time":"2022-01-21T03:44:57Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:generateName":{},"f:labels":{".":{},"f:app":{},"f:pod-template-hash":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"0fa06b5f-d1a9-46a4-8284-f4bd22aeacde\"}":{".":{},"f:apiVersion":{},"f:blockOwnerDeletion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:spec":{"f:containers":{"k:{\"name\":\"nginx\"}":{".":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{".":{},"f:limits":{".":{},"f:cpu":{},"f:memory":{}},"f:requests":{".":{},"f:cpu":{},"f:memory":{}}},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{},"f:volumeMounts":{".":{},"k:{\"mountPath\":\"/home\"}":{".":{},"f:mountPath":{},"f:name":{}}}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:terminationGracePeriodSeconds":{},"f:volumes":{".":{},"k:{\"name\":\"my-home\"}":{".":{},"f:name":{},"f:persistentVolumeClaim":{".":{},"f:claimName":{}}}}}}},{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2022-01-21T03:45:11Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"192.168.224.145\"}":{".":{},"f:ip":{}}},"f:startTime":{}}}}]},"spec":{"volumes":[{"name":"my-home","persistentVolumeClaim":{"claimName":"my-volume"}},{"name":"default-token-jm5k4","secret":{"secretName":"default-token-jm5k4","defaultMode":420}}],"containers":[{"name":"nginx","image":"nginx","resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"250m","memory":"64Mi"}},"volumeMounts":[{"name":"my-home","mountPath":"/home"},{"name":"default-token-jm5k4","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"Always"}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"ip-192-168-253-162.eu-central-1.compute.internal","securityContext":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true,"preemptionPolicy":"PreemptLowerPriority"},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2022-01-21T03:45:03Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2022-01-21T03:45:11Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2022-01-21T03:45:11Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2022-01-21T03:45:03Z"}],"hostIP":"192.168.253.162","podIP":"192.168.224.145","podIPs":[{"ip":"192.168.224.145"}],"startTime":"2022-01-21T03:45:03Z","containerStatuses":[{"name":"nginx","state":{"running":{"startedAt":"2022-01-21T03:45:10Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"nginx:latest","imageID":"docker-pullable://nginx@sha256:0d17b565c37bcbd895e9d92315a05c1c3c9a29f762b011a10c54a66cd53c9b31","containerID":"docker://54f1c1f865b09ffa8252714564d7ce0cb51b47b8fa33eec71b39588cd189822e","started":true}],"qosClass":"Burstable"}}
I0121 03:52:13.650681      22 round_trippers.go:466] curl -v -XPOST  -H "X-Stream-Protocol-Version: v4.channel.k8s.io" -H "X-Stream-Protocol-Version: v3.channel.k8s.io" -H "X-Stream-Protocol-Version: v2.channel.k8s.io" -H "X-Stream-Protocol-Version: channel.k8s.io" -H "User-Agent: kubectl/v1.23.2 (linux/amd64) kubernetes/9d14243" -H "Authorization: Bearer <masked>" 'https://kas.gitlab.com/k8s-proxy/api/v1/namespaces/ttran-dev/pods/nginx-6fff489d7c-8llx6/exec?command=test&command=-d&command=%2Fhome&container=nginx&stderr=true&stdout=true'
I0121 03:52:13.692214      22 round_trippers.go:570] HTTP Statistics: DNSLookup 0 ms Dial 0 ms TLSHandshake 0 ms Duration 41 ms
I0121 03:52:13.692593      22 round_trippers.go:577] Response Headers:
I0121 03:52:13.692765      22 round_trippers.go:580]     Referrer-Policy: no-referrer
I0121 03:52:13.692970      22 round_trippers.go:580]     Content-Length: 273
I0121 03:52:13.693154      22 round_trippers.go:580]     Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
I0121 03:52:13.693318      22 round_trippers.go:580]     Content-Type: text/html; charset=UTF-8
I0121 03:52:13.693667      22 round_trippers.go:466] curl -v -XGET  -H "User-Agent: kubectl/v1.23.2 (linux/amd64) kubernetes/9d14243" -H "Accept: application/json, */*" -H "Authorization: Bearer <masked>" 'https://kas.gitlab.com/k8s-proxy/api/v1/namespaces/ttran-dev/pods/nginx-6fff489d7c-8llx6'
I0121 03:52:13.827563      22 round_trippers.go:570] HTTP Statistics: GetConnection 0 ms ServerProcessing 133 ms Duration 133 ms
I0121 03:52:13.827999      22 round_trippers.go:577] Response Headers:
I0121 03:52:13.828191      22 round_trippers.go:580]     Content-Type: application/json
I0121 03:52:13.828412      22 round_trippers.go:580]     X-Kubernetes-Pf-Flowschema-Uid: e797b352-8a5f-4baf-9291-df57629e57c6
I0121 03:52:13.828606      22 round_trippers.go:580]     Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
I0121 03:52:13.828801      22 round_trippers.go:580]     Cache-Control: no-cache, private
I0121 03:52:13.828988      22 round_trippers.go:580]     Date: Fri, 21 Jan 2022 03:52:12 GMT
I0121 03:52:13.829142      22 round_trippers.go:580]     Via: 2.0 gitlab-agent/v14.5.0/523894a
I0121 03:52:13.829343      22 round_trippers.go:580]     Via: gRPC/1.0 gitlab-kas/v14.5.0/523894a, 1.1 google
I0121 03:52:13.829526      22 round_trippers.go:580]     X-Kubernetes-Pf-Prioritylevel-Uid: bde5d280-854a-4ee8-ae4b-453ac930146b
I0121 03:52:13.829661      22 round_trippers.go:580]     X-Request-Id: 01FSXC6S54X7QJ3RX3A6RQWFZJ
I0121 03:52:13.829938      22 round_trippers.go:580]     Audit-Id: bd0df23c-0a09-4480-9f0d-e6f17247e695
I0121 03:52:13.830380      22 request.go:1181] Response Body: {"kind":"Pod","apiVersion":"v1","metadata":{"name":"nginx-6fff489d7c-8llx6","generateName":"nginx-6fff489d7c-","namespace":"ttran-dev","uid":"f4239bcb-a048-491a-b93e-d7deaee3155a","resourceVersion":"21878774","creationTimestamp":"2022-01-21T03:44:57Z","labels":{"app":"nginx","pod-template-hash":"6fff489d7c"},"annotations":{"kubernetes.io/psp":"eks.privileged"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"nginx-6fff489d7c","uid":"0fa06b5f-d1a9-46a4-8284-f4bd22aeacde","controller":true,"blockOwnerDeletion":true}],"managedFields":[{"manager":"kube-controller-manager","operation":"Update","apiVersion":"v1","time":"2022-01-21T03:44:57Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:generateName":{},"f:labels":{".":{},"f:app":{},"f:pod-template-hash":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"0fa06b5f-d1a9-46a4-8284-f4bd22aeacde\"}":{".":{},"f:apiVersion":{},"f:blockOwnerDeletion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:spec":{"f:containers":{"k:{\"name\":\"nginx\"}":{".":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{".":{},"f:limits":{".":{},"f:cpu":{},"f:memory":{}},"f:requests":{".":{},"f:cpu":{},"f:memory":{}}},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{},"f:volumeMounts":{".":{},"k:{\"mountPath\":\"/home\"}":{".":{},"f:mountPath":{},"f:name":{}}}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:terminationGracePeriodSeconds":{},"f:volumes":{".":{},"k:{\"name\":\"my-home\"}":{".":{},"f:name":{},"f:persistentVolumeClaim":{".":{},"f:claimName":{}}}}}}},{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2022-01-21T03:45:11Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"192.168.224.145\"}":{".":{},"f:ip":{}}},"f:startTime":{}}}}]},"spec":{"volumes":[{"name":"my-home","persistentVolumeClaim":{"claimName":"my-volume"}},{"name":"default-token-jm5k4","secret":{"secretName":"default-token-jm5k4","defaultMode":420}}],"containers":[{"name":"nginx","image":"nginx","resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"250m","memory":"64Mi"}},"volumeMounts":[{"name":"my-home","mountPath":"/home"},{"name":"default-token-jm5k4","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"Always"}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"ip-192-168-253-162.eu-central-1.compute.internal","securityContext":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true,"preemptionPolicy":"PreemptLowerPriority"},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2022-01-21T03:45:03Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2022-01-21T03:45:11Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2022-01-21T03:45:11Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2022-01-21T03:45:03Z"}],"hostIP":"192.168.253.162","podIP":"192.168.224.145","podIPs":[{"ip":"192.168.224.145"}],"startTime":"2022-01-21T03:45:03Z","containerStatuses":[{"name":"nginx","state":{"running":{"startedAt":"2022-01-21T03:45:10Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"nginx:latest","imageID":"docker-pullable://nginx@sha256:0d17b565c37bcbd895e9d92315a05c1c3c9a29f762b011a10c54a66cd53c9b31","containerID":"docker://54f1c1f865b09ffa8252714564d7ce0cb51b47b8fa33eec71b39588cd189822e","started":true}],"qosClass":"Burstable"}}
I0121 03:52:13.831277      22 round_trippers.go:466] curl -v -XPOST  -H "X-Stream-Protocol-Version: v4.channel.k8s.io" -H "X-Stream-Protocol-Version: v3.channel.k8s.io" -H "X-Stream-Protocol-Version: v2.channel.k8s.io" -H "X-Stream-Protocol-Version: channel.k8s.io" -H "User-Agent: kubectl/v1.23.2 (linux/amd64) kubernetes/9d14243" -H "Authorization: Bearer <masked>" 'https://kas.gitlab.com/k8s-proxy/api/v1/namespaces/ttran-dev/pods/nginx-6fff489d7c-8llx6/exec?command=tar&command=-xmf&command=-&command=-C&command=%2F&container=nginx&stderr=true&stdin=true&stdout=true'
I0121 03:52:13.877176      22 round_trippers.go:570] HTTP Statistics: DNSLookup 0 ms Dial 0 ms TLSHandshake 0 ms Duration 45 ms
I0121 03:52:13.877556      22 round_trippers.go:577] Response Headers:
I0121 03:52:13.877751      22 round_trippers.go:580]     Content-Type: text/html; charset=UTF-8
I0121 03:52:13.877954      22 round_trippers.go:580]     Referrer-Policy: no-referrer
I0121 03:52:13.878150      22 round_trippers.go:580]     Content-Length: 273
I0121 03:52:13.878322      22 round_trippers.go:580]     Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
F0121 03:52:13.878623      22 helpers.go:118] error: unable to upgrade connection: <html><head>
<meta http-equiv="content-type" content="text/html;charset=utf-8">
<title>400 Bad Request</title>
</head>
<body text=#000000 bgcolor=#ffffff>
<h1>Error: Bad Request</h1>
<h2>Your client has issued a malformed or illegal request.</h2>
<h2></h2>
</body></html>
goroutine 1 [running]:
k8s.io/kubernetes/vendor/k8s.io/klog/v2.stacks(0x1)
	/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:1038 +0x8a
k8s.io/kubernetes/vendor/k8s.io/klog/v2.(*loggingT).output(0x307e000, 0x3, 0x0, 0xc0005be8c0, 0x2, {0x25f13e7, 0x10}, 0x307eee0, 0x0)
	/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:987 +0x5fd
k8s.io/kubernetes/vendor/k8s.io/klog/v2.(*loggingT).printDepth(0xc0008e2140, 0x134, 0x0, {0x0, 0x0}, 0xc00086a600, {0xc000379af0, 0x1, 0x1})
	/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:735 +0x1ae
k8s.io/kubernetes/vendor/k8s.io/klog/v2.FatalDepth(...)
	/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:1518
k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/util.fatal({0xc0008e2140, 0x134}, 0xc0009d3bc8)
	/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/util/helpers.go:96 +0xc5
k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/util.checkErr({0x1fe8260, 0xc000379aa0}, 0x1e78198)
	/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/util/helpers.go:191 +0x7d7
k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/util.CheckErr(...)
	/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/util/helpers.go:118
k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/cp.NewCmdCp.func2(0xc0007c1180, {0xc0003f0f80, 0x2, 0x8})
	/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/cp/cp.go:154 +0xb6
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).execute(0xc0007c1180, {0xc0003f0f00, 0x8, 0x8})
	/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:860 +0x5f8
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc0003d9680)
	/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:974 +0x3bc
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).Execute(...)
	/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:902
k8s.io/kubernetes/vendor/k8s.io/component-base/cli.run(0xc0003d9680)
	/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/component-base/cli/run.go:146 +0x325
k8s.io/kubernetes/vendor/k8s.io/component-base/cli.RunNoErrOutput(...)
	/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/component-base/cli/run.go:84
main.main()
	_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubectl/kubectl.go:30 +0x1e

goroutine 6 [chan receive]:
k8s.io/kubernetes/vendor/k8s.io/klog/v2.(*loggingT).flushDaemon(0x0)
	/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:1181 +0x6a
created by k8s.io/kubernetes/vendor/k8s.io/klog/v2.init.0
	/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:420 +0xfb

goroutine 8 [select]:
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x0, {0x1fea220, 0xc0008176b0}, 0x1, 0xc00004c5a0)
	/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:167 +0x13b
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0, 0x12a05f200, 0x0, 0x0, 0x0)
	/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x89
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.Until(...)
	/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.Forever(0x0, 0x0)
	/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:81 +0x28
created by k8s.io/kubernetes/vendor/k8s.io/component-base/logs.InitLogs
	/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/component-base/logs/logs.go:179 +0x85

goroutine 24 [select]:
io.(*pipe).Write(0xc0002b23c0, {0x30b0700, 0x0, 0x200})
	/usr/local/go/src/io/pipe.go:94 +0x21a
io.(*PipeWriter).Write(0xc0006c6bd0, {0x30b0700, 0x1986aa0, 0xc000377980})
	/usr/local/go/src/io/pipe.go:163 +0x25
archive/tar.(*Writer).Flush(0xc00009fc00)
	/usr/local/go/src/archive/tar/writer.go:56 +0x86
archive/tar.(*Writer).WriteHeader(0xc00009fc00, 0xc0000117a0)
	/usr/local/go/src/archive/tar/writer.go:68 +0x3e
k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/cp.recursiveTar({{0x1bebcef, 0x7}}, {{0xc000372c50, 0xc000378ce0}}, {{0x7ffc1bb66e7b, 0x7ffc1bb66e7c}}, {{0xc000372c70, 0x1bebd00}}, 0xc0009c3200)
	/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/cp/cp.go:464 +0x395
k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/cp.recursiveTar({{0x1bebcef, 0x0}}, {{0x7ffc1bb66e5e, 0xc0003ee180}}, {{0x7ffc1bb66e7b, 0x1a0c560}}, {{0x7ffc1bb66e7c, 0xc000060690}}, 0x0)
	/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/cp/cp.go:437 +0x8b1
k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/cp.makeTar({{0x7ffc1bb66e5e, 0x200d710}}, {{0x7ffc1bb66e7b, 0xc0007ed880}}, {0x1fe8320, 0xc0003e6850})
	/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/cp/cp.go:410 +0x26f
k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/cp.(*CopyOptions).copyToPod.func1({{0x7ffc1bb66e5e, 0xc0003ee300}}, {{0x7ffc1bb66e7b, 0x51e6d6}}, {0x200d710, 0xc0003e6850})
	/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/cp/cp.go:299 +0xc5
created by k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/cp.(*CopyOptions).copyToPod
	/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/cp/cp.go:297 +0x3b2

goroutine 17 [IO wait]:
internal/poll.runtime_pollWait(0x7f4644463898, 0x72)
	/usr/local/go/src/runtime/netpoll.go:234 +0x89
internal/poll.(*pollDesc).wait(0xc00081e980, 0xc000107000, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00081e980, {0xc000107000, 0x1595, 0x1595})
	/usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a
net.(*netFD).Read(0xc00081e980, {0xc000107000, 0xc000107005, 0x22})
	/usr/local/go/src/net/fd_posix.go:56 +0x29
net.(*conn).Read(0xc0003e61d0, {0xc000107000, 0x64e619, 0xc0000757f8})
	/usr/local/go/src/net/net.go:183 +0x45
crypto/tls.(*atLeastReader).Read(0xc0006c6cd8, {0xc000107000, 0x0, 0x409bcd})
	/usr/local/go/src/crypto/tls/conn.go:777 +0x3d
bytes.(*Buffer).ReadFrom(0xc0007edaf8, {0x1fe8060, 0xc0006c6cd8})
	/usr/local/go/src/bytes/buffer.go:204 +0x98
crypto/tls.(*Conn).readFromUntil(0xc0007ed880, {0x1feba40, 0xc0003e61d0}, 0x461b7b)
	/usr/local/go/src/crypto/tls/conn.go:799 +0xe5
crypto/tls.(*Conn).readRecordOrCCS(0xc0007ed880, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:606 +0x112
crypto/tls.(*Conn).readRecord(...)
	/usr/local/go/src/crypto/tls/conn.go:574
crypto/tls.(*Conn).Read(0xc0007ed880, {0xc000143000, 0x1000, 0xc000075cd8})
	/usr/local/go/src/crypto/tls/conn.go:1277 +0x16f
bufio.(*Reader).Read(0xc0004f3620, {0xc000011458, 0x9, 0xc000075d18})
	/usr/local/go/src/bufio/bufio.go:227 +0x1b4
io.ReadAtLeast({0x1fe7e60, 0xc0004f3620}, {0xc000011458, 0x9, 0x9}, 0x9)
	/usr/local/go/src/io/io.go:328 +0x9a
io.ReadFull(...)
	/usr/local/go/src/io/io.go:347
net/http.http2readFrameHeader({0xc000011458, 0x9, 0x7d0fab}, {0x1fe7e60, 0xc0004f3620})
	/usr/local/go/src/net/http/h2_bundle.go:1555 +0x6e
net/http.(*http2Framer).ReadFrame(0xc000011420)
	/usr/local/go/src/net/http/h2_bundle.go:1813 +0x95
net/http.(*http2clientConnReadLoop).run(0xc000075f98)
	/usr/local/go/src/net/http/h2_bundle.go:8608 +0x130
net/http.(*http2ClientConn).readLoop(0xc0003ee180)
	/usr/local/go/src/net/http/h2_bundle.go:8531 +0x6f
created by net/http.(*http2Transport).newClientConn
	/usr/local/go/src/net/http/h2_bundle.go:7325 +0xb85
section_end:1642737133:step_script
[0Ksection_start:1642737133:cleanup_file_variables
[0K[0K[36;1mCleaning up project directory and file based variables[0;m[0;m
section_end:1642737133:cleanup_file_variables
[0K[31;1mERROR: Job failed: exit code 1
[0;m