<!--Implementation issues are used break-up a large piece of work into small, discrete tasks that can move independently through the build workflow steps. They're typically used to populate a Feature Epic. Once created, an implementation issue is usually refined in order to populate and review the implementation plan and weight. Example workflow: https://about.gitlab.com/handbook/engineering/development/threat-management/planning/diagram.html#plan--> ## Problem Statement [Operational Container Scanning(OCS)](https://docs.gitlab.com/ee/user/clusters/agent/vulnerabilities.html) implemented in [gitlab-agent](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/tree/master) uses [Trivy k8s](https://aquasecurity.github.io/trivy/v0.38/docs/target/kubernetes/) to scan container images in a k8 cluster. The vulnerability report is retrieved by parsing the logs of the Trivy K8s output as implemented in this MR https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/merge_requests/909+. Parsing logs can be flaky especially if the [logs contain errors](https://gitlab.com/gitlab-org/gitlab/-/issues/384365#note_1352823866) or other previously unknown components. Hence, this issue is meant to explore and implement a more robust approach to retrieve the vulnerability report. ## Business Impact This epic was created as a way to solve https://gitlab.com/gitlab-org/gitlab/-/work_items/384365+ which was part of https://gitlab.com/groups/gitlab-org/-/work_items/9870+ epic. As described in the problem statement section OCS was not working reliably since it could not handle errors properly. This epic ensures that OCS is much more reliable and can be used by many customers that requested it. ## Final design During the Architectural Design exploration we investigate various proposals that were discarded for various reasons. This is the list of discarded approaches: * Proposal 2: [Using a configmap to send the Vulnerability report between the scanning pods and the gitlab-agent](https://gitlab.com/groups/gitlab-org/-/epics/11968#note_1677501707) * Proposal 3: [Using a reporter application that gathers the reports from the scanning pods and sends them to Gitlab](https://gitlab.com/groups/gitlab-org/-/epics/11968#note_1677813749) * Proposal 4: [Using HTTP and configmaps](https://gitlab.com/groups/gitlab-org/-/epics/11968#note_1679881433) The final accepted proposal can be found below ### **Proposal 5:** Using a chained configmaps ### Scanning workflow In the final design we use trivy-k8s-wrapper as a scanning pod. This pod will use Trivy to generate a vulnerability report which then will be compressed and transformed. Finally the report is broken in a chain of configMaps. The GitLab agent reads the chain of configMaps in the correct order retrieving this way the vulnerability report and propagating it to the GitLab instance. ```mermaid sequenceDiagram title: Happy flow gitlab agent->>gitlab agent: Attempts to delete all Configmaps <br /> with label `scan: ocs` on gitlab-agent ns note over gitlab agent: This step ensures that we have <br /> no orphan configmaps from previous runs gitlab agent->>Scan Job (ns1): creates gitlab agent-->Scan Job (ns1): watches status Scan Job (ns1)->>Scan Job (ns1): Creates Trivy report Scan Job (ns1)->>Scan Job (ns1): Extract Vulnerabilities <br /> protobuf > gzip > base64 Scan Job (ns1)->>Scan Job (ns1): Deletes any configmaps with <br />label `agent.gitlab.com/ocs-scan: namespace` Scan Job (ns1)->>Configmap (ocs-ns1-agentID-1): Creates first chained Configmap note over Configmap (ocs-ns1-agentID-1): Configmap<br /> data:"..."<br /> annotations<br /> agent.gitlab.com/scan: ocs <br /> agent.gitlab.com/ocs-ns: ns1 <br /> agent.gitlab.com/ocs-next: ocs-ns1-agentID-2 Scan Job (ns1)->>Configmap (ocs-ns1-agentID-2): Creates second chained Configmap note over Configmap (ocs-ns1-agentID-1): Configmap<br /> data:"..."<br /> annotations<br /> agent.gitlab.com/scan: ocs <br /> agent.gitlab.com/ocs-ns: ns1 <br /> agent.gitlab.com/ocs-next: ocs-ns1-agentID-3 Scan Job (ns1)->>Configmap (ocs-ns1-agentID-3): Creates thrid chained Configmap note over Configmap (ocs-ns1-agentID-1): Configmap<br /> data:"..."<br /> annotations<br /> agent.gitlab.com/scan: ocs <br /> agent.gitlab.com/ocs-ns: ns1 Scan Job (ns1)->>Scan Job (ns1): exits 0 Scan Job (ns1)-->>gitlab agent: gets notified gitlab agent->>gitlab agent: verifies exit code of <br /> scanning pod gitlab agent->>Configmap (ocs-ns1-agentID-1): reads gitlab agent->>Configmap (ocs-ns1-agentID-2): reads gitlab agent->>Configmap (ocs-ns1-agentID-3): reads gitlab agent->>gitlab agent: creates payload for <br />creating vulnerabilities gitlab agent->>KAS: Send request to ceate vulnerabilities gitlab agent->>Configmap (ocs-ns1-agentID-1): deletes gitlab agent->>Configmap (ocs-ns1-agentID-2): deletes gitlab agent->>Configmap (ocs-ns1-agentID-3): deletes gitlab agent->>KAS: Resolves vulnerabilities gitlab agent->>gitlab agent: Attempts to delete all Configmaps <br /> with label `scan: ocs` on gitlab-agent ns note over gitlab agent: This step ensures that we have <br /> no orphan configmaps ``` ### Configmaps ### Information about labels We use the following configmap labels: - `agent.gitlab.com/scan: ocs` : Specifies that this is an OCS scan. This label is required in order to delete all configmaps, that contain it for the gitlab-agent namespace. This is an easy way to delete all configmaps for any scanned namespace. - `agent.gitlab.com/ocs-ns: <namespace>`: The scanned namespace connected to this configmap - `agent.gitlab.com/ocs-next: <name>`: The next chained configmap. If there is no such label then that is the last configmap in the chain. - `agent.gitlab.com/trivy-version: <trivy_version>` : the version of Trivy scan tool ### Name convention for Configmaps Configmaps use the following pattern: `ocs-<namespace>-<agentID>-<seqNum>` - `<namespace>` is the scanned namespace connected to this configmap. - `<agentID` is the agentID responsible for initiating the OCS scan. This is required in order to have more than one gitlab-agent running in the same K8S cluster. - `<seqNum>` this number shows the sequence number of this configmap in the chain. Min value is 1. It needs to be a number `>= 1`. ### Limitations We will introduce a limitation on the size of Trivy report. This limitation is `100MB`. This is 10 times bigger than what we currently support. If this limitation is reached in the scanning pod, it will exit with a specific exit code. Exit codes are collected by the OCS module and the reason for failing will be logged. This will enable customers to know if their scan failed due to this limitation. In the future it might be interesting to propagate the reasons why a scan has failed to Gitlab.