Discuss: Component Metadata DB

Problem to solve

groupcomposition analysis needs to store a large amount of metadata about components/packages that might be used as project dependencies, to support the following features out of CI pipelines:

• Container Scanning
• Dependency Scanning
• License Scanning

There's probably too much data for the primary DB. (To be further explored in #375707 (closed).)

We've explored strategies to reduce the amount of data being stored in the primary DB, but there are downsides:

• It increases the complexity of the implementation. See #374901 (closed)
• It creates coupling b/w two datasets maintained by two different groups. #376272 (comment 1135607934) #375796 (comment 1124759240)
  • groupthreat insights is in charge of SBOM ingestion, and they store components detected in project SBOM.
  • groupcomposition analysis is in charge of the aforementioned product categories, and they store component metadata not tied to any project.

Proposal

Evaluate the feasibility of a Component Metadata DB used to support product categories maintained by Composition Analysis.

Links

https://docs.gitlab.com/ee/development/database/multiple_databases.html