Unable to setup Kubernetes agent server using Omnibus 15.8.1-ce.0
I am trying to set up deploying workflow in GitLab CI/CD Docker in Docker runner. I am able to create a Gitlab Agent pod in Kubernetes and to establish connection to my self-hosted instance using Helm Chart.
P.S. My Kubernetes cluster and my Linux Omnibus instance are on different hosts
My changes:
gitlab.rb
gitlab_kas['enable'] = trueNginx virtual host configuration
location /-/kubernetes-agent/ {
proxy_pass http://localhost:8150;
proxy_http_version 1.1;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-NginX-Proxy true;
proxy_set_header Host $host;
proxy_set_header Sec-WebSocket-Protocol $http_sec_websocket_protocol;
proxy_set_header Sec-WebSocket-Extensions $http_sec_websocket_extensions;
proxy_set_header Sec-WebSocket-Key $http_sec_websocket_key;
proxy_set_header Sec-WebSocket-Version $http_sec_websocket_version;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_cache_bypass $http_upgrade;
}
location /-/kubernetes-agent/k8s-proxy/ {
proxy_pass http://localhost:8154;
}My .gitlab/agents/agentName/config.yaml
groups:
- id: infrastructure
observability:
logging:
level: debug
starboard:
cadence: "30 * * * *"DockerInDocker Job logs:
Running with gitlab-runner 15.8.1 (f86890c6)
on My Gitlab Runner M3XNsSoF, system ID: s_a53dfa61fd1b
Preparing the "docker" executor
Using Docker executor with image bitnami/kubectl:1.27.2 ...
Starting service docker:20.10.23-dind ...
Pulling docker image docker:20.10.23-dind ...
Using docker image sha256:efca87bb8a34c0943adb2a99ee00daac9540aaf806a49c60ab90f98bd803e8dd for docker:20.10.23-dind with digest docker@sha256:eb2b932b67488ad8db885efb38e2417810c06a2a64d585ae861d3893e50f2bc8 ...
Waiting for services to be up and running (timeout 30 seconds)...
Pulling docker image bitnami/kubectl:1.27.2 ...
Using docker image sha256:116035bccfe15e109694ad6d4d418d6b42de7aca5c923b525764cad0cad6b39f for bitnami/kubectl:1.27.2 with digest bitnami/kubectl@sha256:ac7532f0f4b609ab306d5f16b2741cf953b0e28e215f8711db1940bdb1c0fc1a ...
Preparing environment
Running on runner-m3xnssof-project-18-concurrent-0 via $SERVER...
Getting source from Git repository 00:01
Fetching changes with git depth set to 20...
Reinitialized existing Git repository in /builds/infrastructure/kube-agent/.git/
Checking out 0aaea9be as develop...
Skipping Git submodules setup
Executing "step_script" stage of the job script 00:01
Using docker image sha256:116035bccfe15e109694ad6d4d418d6b42de7aca5c923b525764cad0cad6b39f for bitnami/kubectl:1.27.2 with digest bitnami/kubectl@sha256:ac7532f0f4b609ab306d5f16b2741cf953b0e28e215f8711db1940bdb1c0fc1a ...
$ kubectl config get-contexts
CURRENT NAME CLUSTER AUTHINFO NAMESPACE
infrastructure/global-ci:ilkov gitlab agent:22
$ kubectl config use-context $KUBE_CONTEXT
Switched to context "infrastructure/global-ci:ilkov".
$ kubectl config view
apiVersion: v1
clusters:
- cluster:
server: https://$HOST/-/kubernetes-agent/k8s-proxy/
name: gitlab
contexts:
- context:
cluster: gitlab
user: agent:22
name: infrastructure/global-ci:ilkov
current-context: infrastructure/global-ci:ilkov
kind: Config
preferences: {}
users:
- name: agent:22
user:
token: REDACTED
$ kubectl get pods -v=10
I0615 22:01:55.048040 30 loader.go:373] Config loaded from file: /builds/infrastructure/kube-agent.tmp/KUBECONFIG
I0615 22:01:55.048563 30 round_trippers.go:466] curl -v -XGET -H "Authorization: Bearer <masked>" -H "Accept: application/json;g=apidiscovery.k8s.io;v=v2beta1;as=APIGroupDiscoveryList,application/json" -H "User-Agent: kubectl/v1.27.2 (linux/amd64) kubernetes/7f6f68f" 'https://$HOST/-/kubernetes-agent/k8s-proxy/api?timeout=32s'
I0615 22:01:55.049759 30 round_trippers.go:495] HTTP Trace: DNS Lookup for $HOST resolved to [{$SERVER_IP_ADDRESS }]
I0615 22:01:55.050163 30 round_trippers.go:510] HTTP Trace: Dial to tcp:$SERVER_IP_ADDRESS:443 succeed
I0615 22:01:55.149512 30 round_trippers.go:553] GET https://$HOST/-/kubernetes-agent/k8s-proxy/api?timeout=32s 404 Not Found in 100 milliseconds
I0615 22:01:55.149534 30 round_trippers.go:570] HTTP Statistics: DNSLookup 0 ms Dial 0 ms TLSHandshake 21 ms ServerProcessing 77 ms Duration 100 ms
I0615 22:01:55.149553 30 round_trippers.go:577] Response Headers:
I0615 22:01:55.149561 30 round_trippers.go:580] X-Kubernetes-Pf-Prioritylevel-Uid: f5652b3d-b281-4021-b063-322c8ab96af1
I0615 22:01:55.149576 30 round_trippers.go:580] Server: nginx
I0615 22:01:55.149583 30 round_trippers.go:580] Date: Thu, 15 Jun 2023 22:01:55 GMT
I0615 22:01:55.149589 30 round_trippers.go:580] Content-Type: text/plain; charset=utf-8
I0615 22:01:55.149595 30 round_trippers.go:580] Audit-Id: 38b83a69-0312-4ff4-9659-58645f9c9e08
I0615 22:01:55.149601 30 round_trippers.go:580] X-Kubernetes-Pf-Flowschema-Uid: ff7e0ec1-f5b4-4250-919f-a9641fae9aa5
I0615 22:01:55.149607 30 round_trippers.go:580] Content-Length: 19
I0615 22:01:55.149613 30 round_trippers.go:580] Cache-Control: no-cache, private
I0615 22:01:55.149619 30 round_trippers.go:580] Via: 2.0 gitlab-agent/v15.8.0/cdcacd91
I0615 22:01:55.149625 30 round_trippers.go:580] Via: gRPC/1.0 gitlab-kas/v15.8.0/cdcacd91
I0615 22:01:55.149631 30 round_trippers.go:580] X-Content-Type-Options: nosniff
I0615 22:01:55.150511 30 request.go:1188] Response Body: 404 page not found
I0615 22:01:55.151073 30 round_trippers.go:466] curl -v -XGET -H "Accept: application/json;g=apidiscovery.k8s.io;v=v2beta1;as=APIGroupDiscoveryList,application/json" -H "User-Agent: kubectl/v1.27.2 (linux/amd64) kubernetes/7f6f68f" -H "Authorization: Bearer <masked>" 'https://$HOST/-/kubernetes-agent/k8s-proxy/apis?timeout=32s'
I0615 22:01:55.177961 30 round_trippers.go:553] GET https://$HOST/-/kubernetes-agent/k8s-proxy/apis?timeout=32s 404 Not Found in 26 milliseconds
I0615 22:01:55.178270 30 round_trippers.go:570] HTTP Statistics: GetConnection 0 ms ServerProcessing 26 ms Duration 26 ms
I0615 22:01:55.178372 30 round_trippers.go:577] Response Headers:
I0615 22:01:55.178472 30 round_trippers.go:580] Audit-Id: 87f84e13-f831-42f7-aaca-f7ce861e2271
I0615 22:01:55.178574 30 round_trippers.go:580] Cache-Control: no-cache, private
I0615 22:01:55.178668 30 round_trippers.go:580] Via: 2.0 gitlab-agent/v15.8.0/cdcacd91
I0615 22:01:55.178758 30 round_trippers.go:580] Via: gRPC/1.0 gitlab-kas/v15.8.0/cdcacd91
I0615 22:01:55.178854 30 round_trippers.go:580] X-Content-Type-Options: nosniff
I0615 22:01:55.178955 30 round_trippers.go:580] X-Kubernetes-Pf-Flowschema-Uid: ff7e0ec1-f5b4-4250-919f-a9641fae9aa5
I0615 22:01:55.179063 30 round_trippers.go:580] X-Kubernetes-Pf-Prioritylevel-Uid: f5652b3d-b281-4021-b063-322c8ab96af1
I0615 22:01:55.179161 30 round_trippers.go:580] Server: nginx
I0615 22:01:55.179242 30 round_trippers.go:580] Date: Thu, 15 Jun 2023 22:01:55 GMT
I0615 22:01:55.179315 30 round_trippers.go:580] Content-Type: text/plain; charset=utf-8
I0615 22:01:55.179399 30 round_trippers.go:580] Content-Length: 19
I0615 22:01:55.179679 30 request.go:1188] Response Body: 404 page not found
E0615 22:01:55.180112 30 memcache.go:265] couldn't get current server API group list: the server could not find the requested resource
I0615 22:01:55.180263 30 cached_discovery.go:120] skipped caching discovery info due to the server could not find the requested resource
I0615 22:01:55.180463 30 round_trippers.go:466] curl -v -XGET -H "Authorization: Bearer <masked>" -H "Accept: application/json;g=apidiscovery.k8s.io;v=v2beta1;as=APIGroupDiscoveryList,application/json" -H "User-Agent: kubectl/v1.27.2 (linux/amd64) kubernetes/7f6f68f" 'https://$HOST/-/kubernetes-agent/k8s-proxy/api?timeout=32s'
I0615 22:01:55.207046 30 round_trippers.go:553] GET https://$HOST/-/kubernetes-agent/k8s-proxy/api?timeout=32s 404 Not Found in 26 milliseconds
I0615 22:01:55.207072 30 round_trippers.go:570] HTTP Statistics: GetConnection 0 ms ServerProcessing 26 ms Duration 26 ms
I0615 22:01:55.207079 30 round_trippers.go:577] Response Headers:
I0615 22:01:55.207086 30 round_trippers.go:580] X-Kubernetes-Pf-Flowschema-Uid: ff7e0ec1-f5b4-4250-919f-a9641fae9aa5
I0615 22:01:55.207092 30 round_trippers.go:580] X-Kubernetes-Pf-Prioritylevel-Uid: f5652b3d-b281-4021-b063-322c8ab96af1
I0615 22:01:55.207099 30 round_trippers.go:580] Server: nginx
I0615 22:01:55.207105 30 round_trippers.go:580] Cache-Control: no-cache, private
I0615 22:01:55.207110 30 round_trippers.go:580] Via: 2.0 gitlab-agent/v15.8.0/cdcacd91
I0615 22:01:55.207117 30 round_trippers.go:580] Via: gRPC/1.0 gitlab-kas/v15.8.0/cdcacd91
I0615 22:01:55.207122 30 round_trippers.go:580] X-Content-Type-Options: nosniff
I0615 22:01:55.207128 30 round_trippers.go:580] Date: Thu, 15 Jun 2023 22:01:55 GMT
I0615 22:01:55.207139 30 round_trippers.go:580] Content-Type: text/plain; charset=utf-8
I0615 22:01:55.207145 30 round_trippers.go:580] Content-Length: 19
I0615 22:01:55.207151 30 round_trippers.go:580] Audit-Id: fbbe2d55-ccbc-46b9-9ba3-c271c86cb022
I0615 22:01:55.207313 30 request.go:1188] Response Body: 404 page not found
I0615 22:01:55.207991 30 round_trippers.go:466] curl -v -XGET -H "Accept: application/json;g=apidiscovery.k8s.io;v=v2beta1;as=APIGroupDiscoveryList,application/json" -H "User-Agent: kubectl/v1.27.2 (linux/amd64) kubernetes/7f6f68f" -H "Authorization: Bearer <masked>" 'https://$HOST/-/kubernetes-agent/k8s-proxy/apis?timeout=32s'
I0615 22:01:55.231939 30 round_trippers.go:553] GET https://$HOST/-/kubernetes-agent/k8s-proxy/apis?timeout=32s 404 Not Found in 23 milliseconds
I0615 22:01:55.231959 30 round_trippers.go:570] HTTP Statistics: GetConnection 0 ms ServerProcessing 23 ms Duration 23 ms
I0615 22:01:55.231975 30 round_trippers.go:577] Response Headers:
I0615 22:01:55.231982 30 round_trippers.go:580] Cache-Control: no-cache, private
I0615 22:01:55.231989 30 round_trippers.go:580] X-Content-Type-Options: nosniff
I0615 22:01:55.231996 30 round_trippers.go:580] X-Kubernetes-Pf-Prioritylevel-Uid: f5652b3d-b281-4021-b063-322c8ab96af1
I0615 22:01:55.232027 30 round_trippers.go:580] Server: nginx
I0615 22:01:55.232033 30 round_trippers.go:580] Date: Thu, 15 Jun 2023 22:01:55 GMT
I0615 22:01:55.232039 30 round_trippers.go:580] Audit-Id: 75851e50-0dd7-46da-9493-a0207e3c7249
I0615 22:01:55.232045 30 round_trippers.go:580] Via: 2.0 gitlab-agent/v15.8.0/cdcacd91
I0615 22:01:55.232051 30 round_trippers.go:580] Via: gRPC/1.0 gitlab-kas/v15.8.0/cdcacd91
I0615 22:01:55.232057 30 round_trippers.go:580] X-Kubernetes-Pf-Flowschema-Uid: ff7e0ec1-f5b4-4250-919f-a9641fae9aa5
I0615 22:01:55.232063 30 round_trippers.go:580] Content-Type: text/plain; charset=utf-8
I0615 22:01:55.232069 30 round_trippers.go:580] Content-Length: 19
I0615 22:01:55.232340 30 request.go:1188] Response Body: 404 page not found
E0615 22:01:55.232578 30 memcache.go:265] couldn't get current server API group list: the server could not find the requested resource
I0615 22:01:55.232587 30 cached_discovery.go:120] skipped caching discovery info due to the server could not find the requested resource
I0615 22:01:55.232601 30 shortcut.go:100] Error loading discovery information: the server could not find the requested resource
I0615 22:01:55.232673 30 round_trippers.go:466] curl -v -XGET -H "Accept: application/json;g=apidiscovery.k8s.io;v=v2beta1;as=APIGroupDiscoveryList,application/json" -H "User-Agent: kubectl/v1.27.2 (linux/amd64) kubernetes/7f6f68f" -H "Authorization: Bearer <masked>" 'https://$HOST/-/kubernetes-agent/k8s-proxy/api?timeout=32s'
I0615 22:01:55.259176 30 round_trippers.go:553] GET https://$HOST/-/kubernetes-agent/k8s-proxy/api?timeout=32s 404 Not Found in 26 milliseconds
I0615 22:01:55.259197 30 round_trippers.go:570] HTTP Statistics: GetConnection 0 ms ServerProcessing 25 ms Duration 26 ms
I0615 22:01:55.259205 30 round_trippers.go:577] Response Headers:
I0615 22:01:55.259212 30 round_trippers.go:580] Via: 2.0 gitlab-agent/v15.8.0/cdcacd91
I0615 22:01:55.259219 30 round_trippers.go:580] Via: gRPC/1.0 gitlab-kas/v15.8.0/cdcacd91
I0615 22:01:55.259225 30 round_trippers.go:580] X-Kubernetes-Pf-Flowschema-Uid: ff7e0ec1-f5b4-4250-919f-a9641fae9aa5
I0615 22:01:55.259231 30 round_trippers.go:580] Content-Type: text/plain; charset=utf-8
I0615 22:01:55.259237 30 round_trippers.go:580] Audit-Id: 96310c2d-44fe-413c-8125-61672f275a31
I0615 22:01:55.259243 30 round_trippers.go:580] Content-Length: 19
I0615 22:01:55.259249 30 round_trippers.go:580] Cache-Control: no-cache, private
I0615 22:01:55.259255 30 round_trippers.go:580] X-Content-Type-Options: nosniff
I0615 22:01:55.259261 30 round_trippers.go:580] X-Kubernetes-Pf-Prioritylevel-Uid: f5652b3d-b281-4021-b063-322c8ab96af1
I0615 22:01:55.259268 30 round_trippers.go:580] Server: nginx
I0615 22:01:55.259274 30 round_trippers.go:580] Date: Thu, 15 Jun 2023 22:01:55 GMT
I0615 22:01:55.259406 30 request.go:1188] Response Body: 404 page not found
I0615 22:01:55.259637 30 round_trippers.go:466] curl -v -XGET -H "Accept: application/json;g=apidiscovery.k8s.io;v=v2beta1;as=APIGroupDiscoveryList,application/json" -H "User-Agent: kubectl/v1.27.2 (linux/amd64) kubernetes/7f6f68f" -H "Authorization: Bearer <masked>" 'https://$HOST/-/kubernetes-agent/k8s-proxy/apis?timeout=32s'
I0615 22:01:55.284265 30 round_trippers.go:553] GET https://$HOST/-/kubernetes-agent/k8s-proxy/apis?timeout=32s 404 Not Found in 24 milliseconds
I0615 22:01:55.284286 30 round_trippers.go:570] HTTP Statistics: GetConnection 0 ms ServerProcessing 23 ms Duration 24 ms
I0615 22:01:55.284295 30 round_trippers.go:577] Response Headers:
I0615 22:01:55.284302 30 round_trippers.go:580] X-Kubernetes-Pf-Prioritylevel-Uid: f5652b3d-b281-4021-b063-322c8ab96af1
I0615 22:01:55.284309 30 round_trippers.go:580] Server: nginx
I0615 22:01:55.284315 30 round_trippers.go:580] Date: Thu, 15 Jun 2023 22:01:55 GMT
I0615 22:01:55.284327 30 round_trippers.go:580] Audit-Id: 59093226-5f1b-4081-be84-939eb6494695
I0615 22:01:55.284333 30 round_trippers.go:580] X-Content-Type-Options: nosniff
I0615 22:01:55.284339 30 round_trippers.go:580] X-Kubernetes-Pf-Flowschema-Uid: ff7e0ec1-f5b4-4250-919f-a9641fae9aa5
I0615 22:01:55.284345 30 round_trippers.go:580] Content-Type: text/plain; charset=utf-8
I0615 22:01:55.284351 30 round_trippers.go:580] Content-Length: 19
I0615 22:01:55.284357 30 round_trippers.go:580] Cache-Control: no-cache, private
I0615 22:01:55.284366 30 round_trippers.go:580] Via: 2.0 gitlab-agent/v15.8.0/cdcacd91
I0615 22:01:55.284373 30 round_trippers.go:580] Via: gRPC/1.0 gitlab-kas/v15.8.0/cdcacd91
I0615 22:01:55.284581 30 request.go:1188] Response Body: 404 page not found
E0615 22:01:55.284873 30 memcache.go:265] couldn't get current server API group list: the server could not find the requested resource
I0615 22:01:55.284882 30 cached_discovery.go:120] skipped caching discovery info due to the server could not find the requested resource
I0615 22:01:55.285023 30 round_trippers.go:466] curl -v -XGET -H "Authorization: Bearer <masked>" -H "Accept: application/json;g=apidiscovery.k8s.io;v=v2beta1;as=APIGroupDiscoveryList,application/json" -H "User-Agent: kubectl/v1.27.2 (linux/amd64) kubernetes/7f6f68f" 'https://$HOST/-/kubernetes-agent/k8s-proxy/api?timeout=32s'
I0615 22:01:55.310457 30 round_trippers.go:553] GET https://$HOST/-/kubernetes-agent/k8s-proxy/api?timeout=32s 404 Not Found in 25 milliseconds
I0615 22:01:55.310473 30 round_trippers.go:570] HTTP Statistics: GetConnection 0 ms ServerProcessing 25 ms Duration 25 ms
I0615 22:01:55.310485 30 round_trippers.go:577] Response Headers:
I0615 22:01:55.310499 30 round_trippers.go:580] X-Content-Type-Options: nosniff
I0615 22:01:55.310505 30 round_trippers.go:580] X-Kubernetes-Pf-Prioritylevel-Uid: f5652b3d-b281-4021-b063-322c8ab96af1
I0615 22:01:55.310512 30 round_trippers.go:580] Date: Thu, 15 Jun 2023 22:01:55 GMT
I0615 22:01:55.310518 30 round_trippers.go:580] Content-Type: text/plain; charset=utf-8
I0615 22:01:55.310524 30 round_trippers.go:580] Content-Length: 19
I0615 22:01:55.310530 30 round_trippers.go:580] Via: 2.0 gitlab-agent/v15.8.0/cdcacd91
I0615 22:01:55.310536 30 round_trippers.go:580] Via: gRPC/1.0 gitlab-kas/v15.8.0/cdcacd91
I0615 22:01:55.310542 30 round_trippers.go:580] X-Kubernetes-Pf-Flowschema-Uid: ff7e0ec1-f5b4-4250-919f-a9641fae9aa5
I0615 22:01:55.310547 30 round_trippers.go:580] Server: nginx
I0615 22:01:55.310553 30 round_trippers.go:580] Audit-Id: dc5e2f9a-46b3-49c8-85d1-977b116b41c6
I0615 22:01:55.310559 30 round_trippers.go:580] Cache-Control: no-cache, private
I0615 22:01:55.310700 30 request.go:1188] Response Body: 404 page not found
I0615 22:01:55.310981 30 round_trippers.go:466] curl -v -XGET -H "Accept: application/json;g=apidiscovery.k8s.io;v=v2beta1;as=APIGroupDiscoveryList,application/json" -H "User-Agent: kubectl/v1.27.2 (linux/amd64) kubernetes/7f6f68f" -H "Authorization: Bearer <masked>" 'https://$HOST/-/kubernetes-agent/k8s-proxy/apis?timeout=32s'
I0615 22:01:55.337227 30 round_trippers.go:553] GET https://$HOST/-/kubernetes-agent/k8s-proxy/apis?timeout=32s 404 Not Found in 26 milliseconds
I0615 22:01:55.337247 30 round_trippers.go:570] HTTP Statistics: GetConnection 0 ms ServerProcessing 25 ms Duration 26 ms
I0615 22:01:55.337254 30 round_trippers.go:577] Response Headers:
I0615 22:01:55.337260 30 round_trippers.go:580] Date: Thu, 15 Jun 2023 22:01:55 GMT
I0615 22:01:55.337272 30 round_trippers.go:580] Cache-Control: no-cache, private
I0615 22:01:55.337278 30 round_trippers.go:580] Via: 2.0 gitlab-agent/v15.8.0/cdcacd91
I0615 22:01:55.337284 30 round_trippers.go:580] Via: gRPC/1.0 gitlab-kas/v15.8.0/cdcacd91
I0615 22:01:55.337290 30 round_trippers.go:580] X-Content-Type-Options: nosniff
I0615 22:01:55.337297 30 round_trippers.go:580] X-Kubernetes-Pf-Prioritylevel-Uid: f5652b3d-b281-4021-b063-322c8ab96af1
I0615 22:01:55.337303 30 round_trippers.go:580] Server: nginx
I0615 22:01:55.337309 30 round_trippers.go:580] Content-Length: 19
I0615 22:01:55.337315 30 round_trippers.go:580] Audit-Id: eb1ceaed-3ca9-47f2-8cf1-79cc8b39c256
I0615 22:01:55.337322 30 round_trippers.go:580] X-Kubernetes-Pf-Flowschema-Uid: ff7e0ec1-f5b4-4250-919f-a9641fae9aa5
I0615 22:01:55.337328 30 round_trippers.go:580] Content-Type: text/plain; charset=utf-8
I0615 22:01:55.337471 30 request.go:1188] Response Body: 404 page not found
E0615 22:01:55.337699 30 memcache.go:265] couldn't get current server API group list: the server could not find the requested resource
I0615 22:01:55.337711 30 cached_discovery.go:120] skipped caching discovery info due to the server could not find the requested resource
I0615 22:01:55.337781 30 round_trippers.go:466] curl -v -XGET -H "Accept: application/json;g=apidiscovery.k8s.io;v=v2beta1;as=APIGroupDiscoveryList,application/json" -H "User-Agent: kubectl/v1.27.2 (linux/amd64) kubernetes/7f6f68f" -H "Authorization: Bearer <masked>" 'https://$HOST/-/kubernetes-agent/k8s-proxy/api?timeout=32s'
I0615 22:01:55.362957 30 round_trippers.go:553] GET https://$HOST/-/kubernetes-agent/k8s-proxy/api?timeout=32s 404 Not Found in 25 milliseconds
I0615 22:01:55.362976 30 round_trippers.go:570] HTTP Statistics: GetConnection 0 ms ServerProcessing 24 ms Duration 25 ms
I0615 22:01:55.362988 30 round_trippers.go:577] Response Headers:
I0615 22:01:55.362996 30 round_trippers.go:580] Content-Type: text/plain; charset=utf-8
I0615 22:01:55.363002 30 round_trippers.go:580] Content-Length: 19
I0615 22:01:55.363009 30 round_trippers.go:580] Via: 2.0 gitlab-agent/v15.8.0/cdcacd91
I0615 22:01:55.363017 30 round_trippers.go:580] Via: gRPC/1.0 gitlab-kas/v15.8.0/cdcacd91
I0615 22:01:55.363027 30 round_trippers.go:580] X-Kubernetes-Pf-Flowschema-Uid: ff7e0ec1-f5b4-4250-919f-a9641fae9aa5
I0615 22:01:55.363033 30 round_trippers.go:580] X-Kubernetes-Pf-Prioritylevel-Uid: f5652b3d-b281-4021-b063-322c8ab96af1
I0615 22:01:55.363043 30 round_trippers.go:580] Server: nginx
I0615 22:01:55.363049 30 round_trippers.go:580] Date: Thu, 15 Jun 2023 22:01:55 GMT
I0615 22:01:55.363061 30 round_trippers.go:580] Audit-Id: 70d3fb41-6a16-438e-bbb0-ef931343dacf
I0615 22:01:55.363067 30 round_trippers.go:580] Cache-Control: no-cache, private
I0615 22:01:55.363073 30 round_trippers.go:580] X-Content-Type-Options: nosniff
I0615 22:01:55.363218 30 request.go:1188] Response Body: 404 page not found
I0615 22:01:55.363537 30 round_trippers.go:466] curl -v -XGET -H "User-Agent: kubectl/v1.27.2 (linux/amd64) kubernetes/7f6f68f" -H "Accept: application/json;g=apidiscovery.k8s.io;v=v2beta1;as=APIGroupDiscoveryList,application/json" -H "Authorization: Bearer <masked>" 'https://$HOST/-/kubernetes-agent/k8s-proxy/apis?timeout=32s'
I0615 22:01:55.388647 30 round_trippers.go:553] GET https://$HOST/-/kubernetes-agent/k8s-proxy/apis?timeout=32s 404 Not Found in 25 milliseconds
I0615 22:01:55.388664 30 round_trippers.go:570] HTTP Statistics: GetConnection 0 ms ServerProcessing 24 ms Duration 25 ms
I0615 22:01:55.388671 30 round_trippers.go:577] Response Headers:
I0615 22:01:55.388679 30 round_trippers.go:580] X-Content-Type-Options: nosniff
I0615 22:01:55.388685 30 round_trippers.go:580] Audit-Id: 9300914f-38fb-42cf-a5bb-0893b9220cf5
I0615 22:01:55.388697 30 round_trippers.go:580] Via: 2.0 gitlab-agent/v15.8.0/cdcacd91
I0615 22:01:55.388709 30 round_trippers.go:580] Via: gRPC/1.0 gitlab-kas/v15.8.0/cdcacd91
I0615 22:01:55.388715 30 round_trippers.go:580] Content-Type: text/plain; charset=utf-8
I0615 22:01:55.388723 30 round_trippers.go:580] Content-Length: 19
I0615 22:01:55.388729 30 round_trippers.go:580] Cache-Control: no-cache, private
I0615 22:01:55.388735 30 round_trippers.go:580] X-Kubernetes-Pf-Flowschema-Uid: ff7e0ec1-f5b4-4250-919f-a9641fae9aa5
I0615 22:01:55.388741 30 round_trippers.go:580] X-Kubernetes-Pf-Prioritylevel-Uid: f5652b3d-b281-4021-b063-322c8ab96af1
I0615 22:01:55.388747 30 round_trippers.go:580] Server: nginx
I0615 22:01:55.388753 30 round_trippers.go:580] Date: Thu, 15 Jun 2023 22:01:55 GMT
I0615 22:01:55.389262 30 request.go:1188] Response Body: 404 page not found
E0615 22:01:55.389474 30 memcache.go:265] couldn't get current server API group list: the server could not find the requested resource
I0615 22:01:55.389483 30 cached_discovery.go:120] skipped caching discovery info due to the server could not find the requested resource
I0615 22:01:55.389986 30 helpers.go:246] server response object: [{
"metadata": {},
"status": "Failure",
"message": "the server could not find the requested resource",
"reason": "NotFound",
"details": {
"causes": [
{
"reason": "UnexpectedServerResponse",
"message": "404 page not found"
}
]
},
"code": 404
}]
Error from server (NotFound): the server could not find the requested resource
Cleaning up project directory and file based variables 00:01
ERROR: Job failed: exit code 1My gitlab agent pod logs:
{"level":"info","time":"2023-06-15T22:01:17.031Z","msg":"Observability endpoint is up","mod_name":"observability","net_network":"tcp","net_address":"[::]:8080"}
{"level":"info","time":"2023-06-15T22:01:17.239Z","msg":"attempting to acquire leader lease gitlab-agent-ilkov/agent-22-lock...\n","agent_id":22}
{"level":"info","time":"2023-06-15T22:01:17.250Z","msg":"successfully acquired lease gitlab-agent-ilkov/agent-22-lock\n","agent_id":22}
{"level":"info","time":"2023-06-15T22:01:17.251Z","msg":"Event occurred","object":{"name":"agent-22-lock","namespace":"gitlab-agent-ilkov"},"fieldPath":"","kind":"Lease","apiVersion":"coordination.k8s.io/v1","type":"Normal","reason":"LeaderElection","message":"ilkov-gitlab-agent-5df7bd84f8-5skvl became leader","agent_id":22}
{"level":"debug","time":"2023-06-15T22:01:17.283Z","msg":"Starboard config is empty, security policies are disabled","mod_name":"starboard_vulnerability","agent_id":22}
{"level":"debug","time":"2023-06-15T22:01:47.314Z","msg":"Starboard config is empty, security policies are disabled","mod_name":"starboard_vulnerability","agent_id":22}
{"level":"debug","time":"2023-06-15T22:01:55.147Z","msg":"Handling GET /-/kubernetes-agent/k8s-proxy/api?timeout=32s","agent_id":22}
{"level":"debug","time":"2023-06-15T22:01:55.169Z","msg":"Handled a connection successfully","mod_name":"reverse_tunnel","agent_id":22}
{"level":"debug","time":"2023-06-15T22:01:55.176Z","msg":"Handling GET /-/kubernetes-agent/k8s-proxy/apis?timeout=32s","agent_id":22}
{"level":"debug","time":"2023-06-15T22:01:55.198Z","msg":"Handled a connection successfully","mod_name":"reverse_tunnel","agent_id":22}
{"level":"debug","time":"2023-06-15T22:01:55.205Z","msg":"Handling GET /-/kubernetes-agent/k8s-proxy/api?timeout=32s","agent_id":22}
{"level":"debug","time":"2023-06-15T22:01:55.227Z","msg":"Handled a connection successfully","mod_name":"reverse_tunnel","agent_id":22}
{"level":"debug","time":"2023-06-15T22:01:55.230Z","msg":"Handling GET /-/kubernetes-agent/k8s-proxy/apis?timeout=32s","agent_id":22}
{"level":"debug","time":"2023-06-15T22:01:55.252Z","msg":"Handled a connection successfully","mod_name":"reverse_tunnel","agent_id":22}
{"level":"debug","time":"2023-06-15T22:01:55.258Z","msg":"Handling GET /-/kubernetes-agent/k8s-proxy/api?timeout=32s","agent_id":22}
{"level":"debug","time":"2023-06-15T22:01:55.279Z","msg":"Handled a connection successfully","mod_name":"reverse_tunnel","agent_id":22}
{"level":"debug","time":"2023-06-15T22:01:55.282Z","msg":"Handling GET /-/kubernetes-agent/k8s-proxy/apis?timeout=32s","agent_id":22}
{"level":"debug","time":"2023-06-15T22:01:55.304Z","msg":"Handled a connection successfully","mod_name":"reverse_tunnel","agent_id":22}
{"level":"debug","time":"2023-06-15T22:01:55.308Z","msg":"Handling GET /-/kubernetes-agent/k8s-proxy/api?timeout=32s","agent_id":22}
{"level":"debug","time":"2023-06-15T22:01:55.331Z","msg":"Handled a connection successfully","mod_name":"reverse_tunnel","agent_id":22}
{"level":"debug","time":"2023-06-15T22:01:55.335Z","msg":"Handling GET /-/kubernetes-agent/k8s-proxy/apis?timeout=32s","agent_id":22}
{"level":"debug","time":"2023-06-15T22:01:55.358Z","msg":"Handled a connection successfully","mod_name":"reverse_tunnel","agent_id":22}
{"level":"debug","time":"2023-06-15T22:01:55.361Z","msg":"Handling GET /-/kubernetes-agent/k8s-proxy/api?timeout=32s","agent_id":22}
{"level":"debug","time":"2023-06-15T22:01:55.383Z","msg":"Handled a connection successfully","mod_name":"reverse_tunnel","agent_id":22}
{"level":"debug","time":"2023-06-15T22:01:55.386Z","msg":"Handling GET /-/kubernetes-agent/k8s-proxy/apis?timeout=32s","agent_id":22}
{"level":"debug","time":"2023-06-15T22:01:55.409Z","msg":"Handled a connection successfully","mod_name":"reverse_tunnel","agent_id":22}
{"level":"debug","time":"2023-06-15T22:02:17.032Z","msg":"Canceled connection","mod_name":"reverse_tunnel","error":"rpc error: code = Canceled desc = context canceled","agent_id":22}
{"level":"debug","time":"2023-06-15T22:02:17.368Z","msg":"Starboard config is empty, security policies are disabled","mod_name":"starboard_vulnerability","agent_id":22}
{"level":"debug","time":"2023-06-15T22:02:47.409Z","msg":"Starboard config is empty, security policies are disabled","mod_name":"starboard_vulnerability","agent_id":22}
{"level":"debug","time":"2023-06-15T22:02:55.146Z","msg":"Canceled connection","mod_name":"reverse_tunnel","error":"rpc error: code = Canceled desc = context canceled","agent_id":22}
{"level":"debug","time":"2023-06-15T22:02:55.146Z","msg":"Canceled connection","mod_name":"reverse_tunnel","error":"rpc error: code = Canceled desc = context canceled","agent_id":22}
{"level":"debug","time":"2023-06-15T22:02:55.146Z","msg":"Canceled connection","mod_name":"reverse_tunnel","error":"rpc error: code = Canceled desc = context canceled","agent_id":22}
{"level":"debug","time":"2023-06-15T22:02:55.170Z","msg":"Canceled connection","mod_name":"reverse_tunnel","error":"rpc error: code = Canceled desc = context canceled","agent_id":22}
{"level":"debug","time":"2023-06-15T22:02:55.228Z","msg":"Canceled connection","mod_name":"reverse_tunnel","error":"rpc error: code = Canceled desc = context canceled","agent_id":22}
{"level":"debug","time":"2023-06-15T22:02:55.252Z","msg":"Canceled connection","mod_name":"reverse_tunnel","error":"rpc error: code = Canceled desc = context canceled","agent_id":22}
{"level":"debug","time":"2023-06-15T22:02:55.280Z","msg":"Canceled connection","mod_name":"reverse_tunnel","error":"rpc error: code = Canceled desc = context canceled","agent_id":22}
{"level":"debug","time":"2023-06-15T22:02:55.305Z","msg":"Canceled connection","mod_name":"reverse_tunnel","error":"rpc error: code = Canceled desc = context canceled","agent_id":22}
{"level":"debug","time":"2023-06-15T22:02:55.358Z","msg":"Canceled connection","mod_name":"reverse_tunnel","error":"rpc error: code = Canceled desc = context canceled","agent_id":22}
{"level":"debug","time":"2023-06-15T22:03:17.459Z","msg":"Starboard config is empty, security policies are disabled","mod_name":"starboard_vulnerability","agent_id":22}
{"level":"debug","time":"2023-06-15T22:03:47.491Z","msg":"Starboard config is empty, security policies are disabled","mod_name":"starboard_vulnerability","agent_id":22}
{"level":"debug","time":"2023-06-15T22:04:17.538Z","msg":"Starboard config is empty, security policies are disabled","mod_name":"starboard_vulnerability","agent_id":22}gitlab-ctl tail gitlab-kas
2023-06-15_21:39:59.71681 {"level":"info","time":"2023-06-15T21:39:59.716Z","msg":"Observability endpoint is up","mod_name":"observability","net_network":"tcp","net_address":"127.0.0.1:8151"}
2023-06-15_21:39:59.71687 {"level":"info","time":"2023-06-15T21:39:59.716Z","msg":"Agentk API endpoint is up","net_network":"tcp","net_address":"127.0.0.1:8150","is_websocket":true}
2023-06-15_21:39:59.71710 {"level":"info","time":"2023-06-15T21:39:59.717Z","msg":"Kubernetes API endpoint is up","mod_name":"kubernetes_api","net_network":"tcp","net_address":"127.0.0.1:8154"}
2023-06-15_21:45:13.28389 {"level":"info","time":"2023-06-15T21:45:13.283Z","msg":"Observability endpoint is up","mod_name":"observability","net_network":"tcp","net_address":"127.0.0.1:8151"}
2023-06-15_21:45:13.28420 {"level":"info","time":"2023-06-15T21:45:13.283Z","msg":"Private API endpoint is up","net_network":"tcp","net_address":"127.0.0.1:8155"}
2023-06-15_21:45:13.28420 {"level":"info","time":"2023-06-15T21:45:13.283Z","msg":"API endpoint is up","net_network":"tcp","net_address":"127.0.0.1:8153"}
2023-06-15_21:45:13.28421 {"level":"info","time":"2023-06-15T21:45:13.283Z","msg":"Agentk API endpoint is up","net_network":"tcp","net_address":"127.0.0.1:8150","is_websocket":true}
2023-06-15_21:45:13.28422 {"level":"info","time":"2023-06-15T21:45:13.283Z","msg":"Kubernetes API endpoint is up","mod_name":"kubernetes_api","net_network":"tcp","net_address":"127.0.0.1:8154"}
2023-06-15_22:01:17.20830 {"level":"info","time":"2023-06-15T22:01:17.207Z","msg":"Config: new commit","grpc_service":"gitlab.agent.agent_configuration.rpc.AgentConfiguration","grpc_method":"GetConfiguration","agent_id":22,"project_id":"infrastructure/global-ci","commit_id":"b093b9fde6ab995cae4e7b683f88790567483552"}Edited by Vladimir Damov