Spike: SCA without CI (pipelineless scans)

Proposal

Today the SCA features (Container Scanning, Dependency Scanning, and License Scanning) are still relying on CI jobs to execute the scans. This means users of GitLab have to adopt the CI before adopting the SCA features.

With the new architecture designed for the SBOM License Scanner and Continuous Vulnerability Scanning, we've drastically reduced our dependency on CI pipeline up to the point where the sole purpose of the CI job is to generate an SBOM. This SBOM provides the list of dependencies that is then leveraged by the Container Scanning, Dependency Scanning, and License Scanning features.

To accellerate the adoption of these security features, we could look at ways to obtain the SBOM without a CI job at all. This would then allow customers to benefit from SCA features right after setting up the repository, and without requiring to configure a CI pipeline. This would be a great opportunity for GitLab to bring new customers with a different adoption path.

Some already suggested approaches:

1. allow users to provide their own SBOM (in the repository or upload via API or UI): this would work for CS, DS, and LC. This SBOM could be generated in their existing CI system for instance.
2. implement SBOM generation logic in the rails platform: this is doable for DS and LC when dependency information is obtained by parsing lockfiles
3. implement a standalone service that generates the SBOM: sames as 2 but also allows to support use cases where obtaining dependency information by running a package manager to generate a parsable file and maybe Container Scanning too.

Things to consider:

1. moving away from CI means that the execution cost is migrating from user (CI minutes) to GitLab (whether it is done in the rails app/sidekick or a service). We might consider a different approach to tracking and billing such cost to customers if needed.
2. the vulnerability management system is currently implemented with the assumption that vulnerabilities are generated within a CI pipelines (excepted Operational Vulnerabilities). There might be additional work in this area to fully decouple our feaures from CI.
   1. Update 2024-02-02: along with OCS, other ways to create vulnerabilities are Mutation.vulnerabilityCreate and CVS.
3. some features like compliance pipelines, scan execution policies, and scan results policies are based on CI pipelines or jobs and would not be compatible with a pipeline-less approach in their current state.