Allow agentk to run vulnerability scans independently

Currently, AgentK depends on the Starboard Operator to run vulnerability scans on its behalf. We have no way of ensuring that this dependent software is present or that these scans actually run. This is undesirable, as we would like for security personnel to be able to configure these scans via scan execution policies so that they run based on configuration which is stored in GitLab.

This MR:

• Addresses gitlab-org/gitlab#349791 (closed) by allowing agentk to run vulnerability scans
• Addresses gitlab-org/gitlab#354944 (closed) by being able to do this without any Starboard resources being installed in the cluster

This MR does not add support for security policies. This will be done in a future iteration (gitlab-org/gitlab#341358 (closed)).

Documentation for this change: gitlab-org/gitlab!85445 (merged)

Steps to setup and validate locally

1. Ensure that KAS is configured on your GDK

2. Setup a local Kubernetes cluster with nothing in it

   1. I recommend k3d

   2. This MR was developed against this version of Kubernetes:

      $ kubectl version
      Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.5", GitCommit:"c285e781331a3785a7f436042c65c5641ce8a9e9", GitTreeState:"clean", BuildDate:"2022-03-16T15:51:05Z", GoVersion:"go1.17.8", Compiler:"gc", Platform:"darwin/amd64"}
      Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1+k3s1", GitCommit:"75dba57f9b1de3ec0403b148c52c348e1dee2a5e", GitTreeState:"clean", BuildDate:"2021-05-21T16:12:29Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"linux/amd64"}

3. In GitLab, create a new project that is licensed with GitLab Ultimate

4. Create an agent configuration file with these contents:

   starboard:
     cadence: '0 * * * *'
     vulnerability_report:
       namespaces:
         - default
   
   observability:
     logging:
       level: debug

5. In your current project, register the agent with GitLab

6. Copy the agent token and save it locally in your gitlab-agent directory:

   echo -n '<token>' > token.txt

7. Add a KAS configuration file (cfg.yaml):

   gitlab:
     address: http://gdk.local:3000
     authentication_secret_file: /path/to/gitlab-development-kit/gitlab/.gitlab_kas_secret
   agent:
     listen:
       network: tcp
       address: 127.0.0.1:8150
       websocket: false
     gitops:
       max_number_of_files: 5

8. Stop the GDK KAS:

   gdk stop gitlab-k8s-agent

9. Run KAS from source:

   go run ./cmd/kas/main.go --configuration-file="$(pwd)/cfg.yaml"

10. Create a resource to scan:

    kubectl create deployment ubuntu --image ubuntu:18.04
    deployment.apps/ubuntu created

11. Run agentk:

    The scan should begin within a minute.

    go run ./cmd/agentk/main.go --kas-address='grpc://127.0.0.1:8150' --token-file="$(pwd)/token.txt"

    Example logs:

    {"level":"info","time":"2022-04-18T13:59:19.703-0500","msg":"Observability endpoint is up","mod_name":"observability","net_network":"tcp","net_address":"[::]:8080"}
    {"level":"debug","time":"2022-04-18T14:00:00.000-0500","msg":"Enumerating scannable workloads","mod_name":"starboard_vulnerability"}
    {"level":"debug","time":"2022-04-18T14:00:00.008-0500","msg":"skipping pod \"ubuntu-694fcd4fb-6f8nn\" controlled by built-in workload: ReplicaSet/ubuntu-694fcd4fb","mod_name":"starboard_vulnerability"}
    {"level":"debug","time":"2022-04-18T14:00:00.042-0500","msg":"Found 1 workloads to scan","mod_name":"starboard_vulnerability"}
    {"level":"debug","time":"2022-04-18T14:00:00.042-0500","msg":"Scanning workload: default/ReplicaSet/ubuntu-694fcd4fb","mod_name":"starboard_vulnerability"}
    {"level":"info","time":"2022-04-18T14:00:08.528-0500","msg":"Creating vulnerabilities in GitLab","mod_name":"starboard_vulnerability","report_name":"replicaset-ubuntu-694fcd4fb-ubuntu","k8s_uid":"","vulnerabilities_count":35}
    {"level":"info","time":"2022-04-18T14:00:23.742-0500","msg":"Resolving no longer detected vulnerabilities in GitLab","mod_name":"starboard_vulnerability","report_name":"replicaset-ubuntu-694fcd4fb-ubuntu","k8s_uid":""}

12. Return to your GitLab project and navigate to Security & Compliance -> Vulnerability Report -> Operational Vulnerabilities. The vulnerabilities from the scan should be present.

    

Things that don't work as expected or could be better

Currently, the namespace and service account used by the scanner are hard-coded to gitlab-agent, which means that this will only work with the default agent installation. We should make this configurable, or see if the agent can auto-detect the namespace / service account that it is running in / as. gitlab-org/gitlab#361972 (closed)