Agent connection times out

On a brand new deployment, the Agent gets stuck in an error state (although its pod shows as healthy) and stopped responding to requests coming from the CI/CD tunnel.

Sample pipeline: https://gitlab.com/gitlab-com/gl-security/security-operations/infrastructure-security/k8s-workloads/-/jobs/1716706776

Error:

Starting deploy...
Get "https://kas.gitlab.com/k8s-proxy/version?timeout=32s": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
Error from agent logs
➜ k logs -f gitlab-agent-c6d5bcf47-j2rbz
+ kubectl logs -f gitlab-agent-c6d5bcf47-j2rbz
{"level":"info","time":"2021-10-26T14:05:20.983Z","msg":"Feature status change","feature_name":"tunnel","feature_status":true}
{"level":"info","time":"2021-10-26T14:05:20.984Z","msg":"Observability endpoint is up","mod_name":"observability","net_network":"tcp","net_address":"[::]:8080"}
{"level":"info","time":"2021-10-26T14:05:21.137Z","msg":"Starting synchronization worker","mod_name":"gitops","project_id":"gitlab-com/gl-security/security-operations/infrastructure-security/k8s-workloads"}
{"level":"error","time":"2021-10-26T14:05:22.287Z","msg":"Failed to decode GitOps objects","mod_name":"gitops","project_id":"gitlab-com/gl-security/security-operations/infrastructure-security/k8s-workloads","agent_id":1540,"error":"[unable to decode \".gitlab-ci.yml\": Object 'Kind' is missing in '{\".k8s_build\":<content of gitlab-ci>],\"repo\":\"https://github.com/pre-commit/pre-commit-hooks\",\"rev\":\"v3.3.0\"},{\"hooks\":[{\"id\":\"prettier\"}],\"repo\":\"https://github.com/pre-commit/mirrors-prettier/\",\"rev\":\"v2.2.0\"},{\"hooks\":[{\"id\":\"terraform_fmt\"},{\"id\":\"terraform_validate\"}],\"repo\":\"git://github.com/antonbabenko/pre-commit-terraform\",\"rev\":\"v1.45.0\"},{\"hooks\":[{\"id\":\"checkov\"}],\"repo\":\"https://github.com/bridgecrewio/checkov.git\",\"rev\":\"1.0.674\"}]}', unable to decode \".prettierrc.json\": Object 'Kind' is missing in '{\n  \"arrowParens\": \"always\",\n  \"bracketSpacing\": false,\n  \"jsxBracketSameLine\": false,\n  \"printWidth\": 120,\n  \"semi\": true,\n  \"singleQuote\": true,\n  \"tabWidth\": 2,\n  \"trailingComma\": \"es5\",\n  \"useTabs\": false\n}', unable to decode \"cartography/deployment/elastic-ingestor/base/es-index.json\": Object 'Kind' is missing in '<content of a json file?', unable to decode \"cartography/deployment/elastic-ingestor/docker/queries.json\": json: cannot unmarshal array into Go value of type unstructured.detector]","

From the logs above, why is the agent trying to parse every file contained in the repo associated with the cluster?

Edited by Marco Lancini (GitLab)