Replace the default container scanning engine of Clair and Klar with Trivy
### Release Notes
### Purpose
To create a new container scanning tool, based on the [gcs prototype](https://gitlab.com/caneldem/gcs/) written by @caneldem, that will be used to replace [Klar](https://gitlab.com/gitlab-org/security-products/analyzers/klar), the current container scanning tool.
Apart from the language change, the main goal is to replace the underlying engine, [clair](https://github.com/quay/clair), with [trivy](https://github.com/aquasecurity/trivy).
The new container scanning tool is to be released in the same manner as the existing one, a Docker image hosted at `registry.gitlab.com/gitlab-org/security-products/analyzers/`, and must be a drop-in replacement for the current one[^1].
For GitLab release %14.0, the [`Container-Scanning.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/Container-Scanning.gitlab-ci.yml) template will be updated to use `CS_MAJOR_VERSION: 4` and possibly use a different image name to avoid confusion.
#### Rationale
As part of the [Vulnerability Scans against Running Containers](https://gitlab.com/groups/gitlab-org/-/epics/3410) epic, we had to identify a container scanning tool for perform the task. While Klar, the current scanner, could be used, a [research spike to evaluate Trivy](https://gitlab.com/gitlab-org/gitlab/-/issues/270888) determined that this tool would be easier to integrate and consume fewer resources during the scan.
### Scope
This Epic must deliver:
1. [x] a new project (pick a name) under https://gitlab.com/gitlab-org/security-products/analyzers. Initial version to contain the [PoC code](https://gitlab.com/caneldem/gcs/) with appropriate security scanners enabled and test reports. Ensure the new [project is part of the product](https://about.gitlab.com/handbook/engineering/merge-request-rate/#projects-that-are-part-of-the-product).
1. [x] the identified [Gap requirements](https://gitlab.com/gitlab-org/gitlab/-/issues/299137#gap-requirements).
1. [x] a working image at `registry.gitlab.com/gitlab-org/security-products/analyzers/gcs`
1. [x] documentation updates containing:
1. [x] how to enable the v4 scanner (gcs) before it's made the default
1. [x] how to upgrade from the v3 (klar) to the v4 (gcs) scanner, including any configuration breaking changes
1. [x] how to roll-back to the v3 scanner the v4 scanner is the default
1. [x] internal documentation updates, including:
1. [x] https://about.gitlab.com/handbook/engineering/development/secure/tech-docs/severity-levels/
1. [x] https://about.gitlab.com/handbook/engineering/development/secure/#skills
1. [x] Improved documentation regarding how to use vulnerability allow listing
1. [x] the capability to run without requiring use of the root user
1. [x] a GitLab copy of the Trivy source code
Follow-on work is tracked in a [separate epic](https://gitlab.com/groups/gitlab-org/-/epics/5462)
#### Testing
Identify GitLab projects to use the new Container Scanning tool before the %14.0 release. These will be used to ensure there are no major behaviour changes or usability issues originating from the change.
Because analyzers are used as a CI template, it's not practical to use a feature flag. Instead, chosen projects will have their configuration overridden to use v4 (i.e. `gcs`) as soon as it's available:
- TBD
### Release timeline
1. %13.10: first release of v4 scanner. Testing with selected projects to commence.
1. %13.11: bug fixes, including critical behaviour changes. Testing expanded and continues.
1. %14.0: container scanning template is updated to use v4 scanner.
### Footnotes
[^1]: for an unmodified use of the [`Container-Scanning.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/Container-Scanning.gitlab-ci.yml) template and with the exception of Clair-specific environment variables such as [`KLAR_TRACE` and `CLAIR_TRACE`](https://gitlab.com/gitlab-org/gitlab/-/issues/299137#note_503603481).
/cc @kkwentus1 @sam.white @gitlab-org/threat-management/defend/container-security/backend
epic