KAS Pipeline cannot send commands to kubernetes cluster, Starboard Vulnerability Config returns 404
Hello,
we are using a self-hosted Gitlab 16.1.3 and a K8S Cluster with 16.1.3 Agent on it. The connection itself is working fine, but when I try to run kubectl commands in the pipeline, we are getting this:
$ kubectl create namespace "$KUBE_NAMESPACE" --dry-run=client -o yaml | kubectl apply -f -v=8 -
Error from server (NotFound): the server could not find the requested resource
In the gitlab logs I see:
gitlab-web-1 | {"time":"2023-07-17T11:11:38.710Z","severity":"INFO","duration_s":0.00081,"db_duration_s":0.0,"view_duration_s":0.00081,"status":404,"method":"GET","path":"/api/v4/internal/kubernetes/modules/starboard_vulnerability/policies_configuration","params":[],"host":"...","remote_ip":"172.23.0.4, 127.0.0.1","ua":"gitlab-kas/v16.1.3/b455b15c","route":"/api/:version/*path","queue_duration_s":0.029892,"redis_calls":7,"redis_duration_s":0.001601,"redis_read_bytes":208,"redis_write_bytes":1177,"redis_feature_flag_calls":7,"redis_feature_flag_duration_s":0.001601,"redis_feature_flag_read_bytes":208,"redis_feature_flag_write_bytes":1177,"db_count":4,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":4,"db_main_count":4,"db_ci_count":0,"db_main_replica_count":0,"db_ci_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_ci_cached_count":0,"db_main_replica_cached_count":0,"db_ci_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_ci_wal_count":0,"db_main_replica_wal_count":0,"db_ci_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_ci_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_ci_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.006,"db_main_duration_s":0.006,"db_ci_duration_s":0.0,"db_main_replica_duration_s":0.0,"db_ci_replica_duration_s":0.0,"cpu_s":0.026648,"mem_objects":8590,"mem_bytes":1197767,"mem_mallocs":2973,"mem_total_bytes":1541367,"pid":706,"worker_id":"puma_10","rate_limiting_gates":[],"correlation_id":"01H5HRRZ5KWP504JWY0R071YBY","meta.caller_id":"* /api/:version/*path","meta.remote_ip":"172.23.0.4","meta.feature_category":"not_owned","meta.client_id":"ip/172.23.0.4","request_urgency":"default","target_duration_s":1,"response_bytes":25}
gitlab-web-1 | {"content_type":"application/json","correlation_id":"01H5HRRZ5KWP504JWY0R071YBY","duration_ms":35,"host":"...","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","route":"^/api/","status":404,"system":"http","time":"2023-07-17T11:11:38Z","ttfb_ms":35,"uri":"/api/v4/internal/kubernetes/modules/starboard_vulnerability/policies_configuration","user_agent":"gitlab-kas/v16.1.3/b455b15c","written_bytes":25}
gitlab-web-1 | 172.23.0.4 - - [17/Jul/2023:11:11:38 +0000] "GET /api/v4/internal/kubernetes/modules/starboard_vulnerability/policies_configuration HTTP/1.1" 404 25 "" "gitlab-kas/v16.1.3/b455b15c" -
In the agent I see:
{"level":"debug","time":"2023-07-17T11:14:38.918Z","msg":"ContainerScanning config is empty, security policies are disabled","mod_name":"starboard_vulnerability","agent_id":13}
In the nginx reverse proxy I see:
[17/Jul/2023:11:08:34 +0000] - 404 404 - GET https kas.ourGitlabDomain.de "/k8s-proxy/openapi/v2?timeout=32s" [Client 141.95.74.75] [Length 19] [Gzip -] [Sent-to gitlab-web-1] "kubectl/v1.23.7 (linux/amd64) kubernetes/42c05a5" "-"
[17/Jul/2023:11:08:34 +0000] - 404 404 - GET https kas.ourGitlabDomain.de "/k8s-proxy/swagger-2.0.0.pb-v1?timeout=32s" [Client 141.95.74.75] [Length 19] [Gzip -] [Sent-to gitlab-web-1] "kubectl/v1.23.7 (linux/amd64) kubernetes/42c05a5" "-"
And when I call GET /api/v4/internal/kubernetes/modules/starboard_vulnerability/policies_configuration manually, I'm getting error: 404 Not found
as response.
Has anyone an idea what this could be?
Edited by Sascha Wolff