OCS with Trivy 52.2 requires more Cluster level permissions
Problem statement
trivy-k8s-wrapper
v0.3.1 upgrades Trivy from 0.49.1
to 0.52.2
. However, running the OCS pod (i.e. the trivy-k8s-wrapper pod) in the gitlab agent fails.
Full error log
{"level":"info","message":"Trivy wrapper image initialized with","gitlab agent namespace":"dev-ga","workloads":"Pod,ReplicaSet,ReplicationController,StatefulSet,DaemonSet,CronJob,Job","namespace"
:"test","gitlab agend id":"1103537","timeout-minutes":15}
{"level":"info","message":"Executing","cmd":"/home/gitlab/trivy -v"}
{"level":"info","message":"Trivy version information","version":"0.52.2"}
Trivy scan: exit status 1
{"level":"info","message":"Executing","cmd":"/home/gitlab/trivy k8s --include-kinds Pod,ReplicaSet,ReplicationController,StatefulSet,DaemonSet,CronJob,Job --report=summary --scanners=vuln --disab
le-node-collector --db-repository registry.gitlab.com/gitlab-org/security-products/dependencies/trivy-db-glad --include-namespaces test --format json --output result.json"}
{"level":"error","message":"Trivy scan","error":"exit status 1"}
{"level":"error","message":"","stderr":"2024-06-28T11:29:07Z\tINFO\tAdding schema version to the DB repository for backward compatibility\trepository=\"registry.gitlab.com/gitlab-org/security-pro
ducts/dependencies/trivy-db-glad:2\"\n2024-06-28T11:29:08Z\tERROR\tUnable to list resources\terror=\"failed listing resources for gvr: /v1, Resource=services - services is forbidden: User "sys
tem:serviceaccount:dev-ga:dev-agent-gitlab-agent-ocs-scanning-pod-sa " cannot list resource "services " in API group " " at the cluster scope\"\n2024-06-28T11:29:08Z\tERROR\tUnable to
list resources\terror=\"failed listing resources for gvr: /v1, Resource=configmaps - configmaps is forbidden: User "system:serviceaccount:dev-ga:dev-agent-gitlab-agent-ocs-scanning-pod-sa "
cannot list resource "configmaps " in API group " " at the cluster scope\"\n2024-06-28T11:29:08Z\tERROR\tUnable to list resources\terror=\"failed listing resources for gvr: rbac.author
ization.k8s.io/v1, Resource=roles - roles.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:dev-ga:dev-agent-gitlab-agent-ocs-scanning-pod-sa " cannot list resource "rol
es " in API group "rbac.authorization.k8s.io " at the cluster scope\"\n2024-06-28T11:29:08Z\tERROR\tUnable to list resources\terror=\"failed listing resources for gvr: rbac.authorization.k
8s.io/v1, Resource=rolebindings - rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:dev-ga:dev-agent-gitlab-agent-ocs-scanning-pod-sa " cannot list resource \\
\"rolebindings " in API group "rbac.authorization.k8s.io " at the cluster scope\"\n2024-06-28T11:29:08Z\tERROR\tUnable to list resources\terror=\"failed listing resources for gvr: networki
ng.k8s.io/v1, Resource=networkpolicies - networkpolicies.networking.k8s.io is forbidden: User "system:serviceaccount:dev-ga:dev-agent-gitlab-agent-ocs-scanning-pod-sa " cannot list resource
"networkpolicies " in API group "networking.k8s.io " at the cluster scope\"\n2024-06-28T11:29:08Z\tERROR\tUnable to list resources\terror=\"failed listing resources for gvr: networking.
k8s.io/v1, Resource=ingresses - ingresses.networking.k8s.io is forbidden: User "system:serviceaccount:dev-ga:dev-agent-gitlab-agent-ocs-scanning-pod-sa " cannot list resource "ingresses\\
\" in API group "networking.k8s.io " at the cluster scope\"\n2024-06-28T11:29:08Z\tERROR\tUnable to list resources\terror=\"failed listing resources for gvr: /v1, Resource=resourcequotas - r
esourcequotas is forbidden: User "system:serviceaccount:dev-ga:dev-agent-gitlab-agent-ocs-scanning-pod-sa " cannot list resource "resourcequotas " in API group " " at the cluster s
cope\"\n2024-06-28T11:29:09Z\tERROR\tUnable to list resources\terror=\"failed listing resources for gvr: /v1, Resource=limitranges - limitranges is forbidden: User "system:serviceaccount:dev-g
a:dev-agent-gitlab-agent-ocs-scanning-pod-sa " cannot list resource "limitranges " in API group " " at the cluster scope\"\n2024-06-28T11:29:09Z\tERROR\tUnable to list resources\terro
r=\"failed listing resources for gvr: rbac.authorization.k8s.io/v1, Resource=clusterroles - clusterroles.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:dev-ga:dev-agent-gi
tlab-agent-ocs-scanning-pod-sa " cannot list resource "clusterroles " in API group "rbac.authorization.k8s.io " at the cluster scope\"\n2024-06-28T11:29:09Z\tERROR\tUnable to list res
ources\terror=\"failed listing resources for gvr: rbac.authorization.k8s.io/v1, Resource=clusterrolebindings - clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:servicea
ccount:dev-ga:dev-agent-gitlab-agent-ocs-scanning-pod-sa " cannot list resource "clusterrolebindings " in API group "rbac.authorization.k8s.io " at the cluster scope\"\n2024-06-28T11:
29:09Z\tERROR\tUnable to list resources\terror=\"failed listing resources for gvr: /v1, Resource=nodes - nodes is forbidden: User "system:serviceaccount:dev-ga:dev-agent-gitlab-agent-ocs-scann
ing-pod-sa " cannot list resource "nodes " in API group " " at the cluster scope\"\n2024-06-28T11:29:09Z\tFATAL\tFatal error\tget k8s artifacts error: nodes is forbidden: User \"syste
m:serviceaccount:dev-ga:dev-agent-gitlab-agent-ocs-scanning-pod-sa\" cannot list resource \"nodes\" in API group \"\" at the cluster scope\n","error":"exit status 1"}
The reason is that the OCS service account is missing the following permissions:
cannot list resource "services " in API group " " at the cluster scope
cannot list resource "configmaps " in API group " " at the cluster
cannot list resource "roles " in API group "rbac.authorization.k8s.io "
cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io
cannot list resource "networkpolicies" in API group "networking.k8s.io "
cannot list resource "ingresses" in API group "networking.k8s.io "
cannot list resource "resourcequotas " in API group " " at the cluster scope
cannot list resource "limitranges " in API group " " at the cluster scope
cannot list resource "clusterroles " in API group "rbac.authorization.k8s.io " at the cluster scope
cannot list resource "clusterrolebindings " in API group "rbac.authorization.k8s.io " at the cluster scope
cannot list resource "nodes " in API group " " at the cluster scope
cannot list resource "nodes\" in API group "\" at the cluster scope
Proposal
Add the required permissions to the OCS service account in the gitlab-agent helm chart.
Verification
After installing the new helm chart we should make sure that the following commands return yes
k auth can-i list services --as=system:serviceaccount:gitlab-agent:gitlab-agent-ocs-scanning-pod-sa
k auth can-i list configmaps --as=system:serviceaccount:gitlab-agent:gitlab-agent-ocs-scanning-pod-sa
k auth can-i list resourcequotas --as=system:serviceaccount:gitlab-agent:gitlab-agent-ocs-scanning-pod-sa
k auth can-i list limitranges --as=system:serviceaccount:gitlab-agent:gitlab-agent-ocs-scanning-pod-sa
k auth can-i list nodes --as=system:serviceaccount:gitlab-agent:gitlab-agent-ocs-scanning-pod-sa
k auth can-i list rolebindings --as=system:serviceaccount:gitlab-agent:gitlab-agent-ocs-scanning-pod-sa
k auth can-i list clusterroles --as=system:serviceaccount:gitlab-agent:gitlab-agent-ocs-scanning-pod-sa
k auth can-i list clusterrolebindings --as=system:serviceaccount:gitlab-agent:gitlab-agent-ocs-scanning-pod-sa
k auth can-i list networkpolicies --as=system:serviceaccount:gitlab-agent:gitlab-agent-ocs-scanning-pod-sa
k auth can-i list ingresses --as=system:serviceaccount:gitlab-agent:gitlab-agent-ocs-scanning-pod-sa
We should run manually OCS with trivy-k8s-wrapper v0.3.1
and make a successful scan.
Relates to
- 17.2 Secure:Composition Analysis Planning Issue (#451087 - closed)
- Update trivy to 0.52.2 (gitlab-org/security-products/analyzers/trivy-k8s-wrapper!33 - merged)
Implementation plan
-
Update Helm chart with new permissions -
Verify locally that the new permissions are correct -
Verify locally that OCS works with the latest trivy-k8s-wrapper image -
Update gitlab-agent code to use the latest trivy-k8s-wrapper image. Make sure that this change will be released along with the helm chart.
Edited by Nick Ilieskou