Engineering DRI: @zmartins ## Background context and Problems to Solve Please reference the [parent epic](https://gitlab.com/groups/gitlab-org/-/epics/7886 "Continuous vulnerability scans") for context and details. This epic specifically deals with providing the backend APIs for the Merge Request, Dependency List, Vulnerability Report, and License Compliance pages in the UI as shown in this [high level flow diagram](https://gitlab.com/groups/gitlab-org/-/epics/7886#proposed-architecture-high-level-concept "Continuous vulnerability scans"). ## Proposal (Requirements) 1. Backend APIs that support the following UI elements will be updated if necessary to display information from the database. 1. License Compliance and Security Scanning merge request widgets - uses license scanning report data 2. ~~Security tab of the Pipeline page~~ - uses vulnerability finding data 3. Dependency List page - uses dependency scanning report data 4. ~~Vulnerability Report page~~ - uses vulnerability finding data 5. License Compliance page - uses license scanning report data 2. Where relevant, database information for **licenses** will be displayed **in addition to** parsing through artifact files. This is to allow us to support the `license-finder` analyzer until it is formally deprecated and removed (likely in %16.0). License information from both will be very simply joined together with no requirement for de-duplication. 3. Frontend UX and functionality is not expected to change as part of this effort. 4. Data source need to be modified in such a way as to supply the same information whether it's for UI elements or for API endpoints. ### Current implementation for retrieving dependency list data Dependency List endpoints query DependencyListService for dependencies for a particular pipeline. The service is responsible for decorating the dependencies with graph, license, and vulnerability finding data. The data flow looks something like: 1. [get dependency scanning reports](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/models/ee/ci/pipeline.rb#L123) for the given pipeline 2. for each dependency in the reports above [add dependency](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/gitlab/ci/reports/dependency_list/report.rb#L23) to DependencyListReport - the [graph data](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/gitlab/ci/reports/dependency_list/report.rb#L72) for that dependency is processed and added as well 3. [fetch vulnerability findings](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/gitlab/ci/parsers/security/dependency_list.rb#L35) for project - [append each finding](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/gitlab/ci/reports/dependency_list/report.rb#L29) to DependencyListReport using `vulnerability.location` to match to existing dependencies 4. [get license scanning reports](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/models/ee/ci/pipeline.rb#L126) for the pipeline - [apply each license](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/gitlab/ci/reports/dependency_list/report.rb#L37) to matching dependency in DependencyListReport DependencyListReport is responsible for deduplication and extraction of path data. - deduplication is done via deriving a composite key: https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/gitlab/ci/reports/dependency_list/dependency.rb#L30 DependencyListService is responsible for filtering, sorting, and paginating report data. It also exposes filtering constants to callers: . ## Non-functional requirements ### Feature flag Add a feature flag to change the source of dependency data from the security reports to the database. ### API The changes must happen at an abstraction level sufficient to ensure that the same data is provided for components. ### Documentation Documentation needs to change to reflect the changed data source from security reports to the database. ## Implementation plan ### Phase 1 - [x] Add `Sbom::DependenciesFinder` which returns dependency data from the SBoM tables - [x] Add a [`Project.dependencies`](https://gitlab.com/gitlab-org/gitlab/-/blob/64155898e65f1648e36a6719e3b4d05d3fa39770/ee/app/graphql/ee/types/project_type.rb#L253) resource to GraphQL which retrieves the dependencies for the project using `Sbom::DependenciesFinder` - [ ] Add a feature flag to the frontend which swaps the data source for the dependency list between `DependenciesController` and the new GraphQL resource ### Phase 2 - [ ] Update `Project.dependencies` to retrieve license information ### Phase 3 - [ ] Update `Project.dependencies` to retrieve vulnerability information - [ ] Release the feature to production - [ ] Remove feature flag