Add `Query.project.dependencies` to GraphQL
What does this MR do and why?
Describe in detail what your merge request does and why.
This MR adds a new GraphQL query to retrieve the dependencies used by a project. This query will serve as an eventual replacement for dependencies.json and the dependencies API. Currently, it does not have complete feature-parity with these APIs. This will be implemented in later iterations.
The goal of this change is to migrate to a new data source. Currently, dependencies are retrieved by parsing JSON CI job artifacts. Sbom::DependenciesFinder retrieves information from the database only. This performs better, and will enable large-scale features such as the group dependency list.
💾 Database
Sbom::DependenciesFinder introduces 4 new queries. These explain plans are generated locally since the queries are dependent on each other and this makes it not feasible to test them in database lab.
Find Sbom::Occurrences (page 2, page size 100)
SELECT
sbom_occurrences.*
FROM
sbom_occurrences
WHERE
sbom_occurrences.project_id = 27 AND
sbom_occurrences.id > 711
ORDER BY
sbom_occurrences.id ASC
LIMIT 101; QUERY PLAN
-------------------------------------------------------------------------------------------------------------------------------
Limit (cost=0.28..10.69 rows=101 width=121)
-> Index Scan using index_sbom_occurrences_on_project_id_and_id on sbom_occurrences (cost=0.28..17.29 rows=165 width=121)
Index Cond: ((project_id = 27) AND (id > 711))
(3 rows)
Time: 0.378 msPreloading Sbom::Component association
SELECT
sbom_components.*
FROM
sbom_components
WHERE
sbom_components.id IN (
29,
660,
661,
37,
38,
664,
665,
666,
667,
668,
39,
670,
40,
672,
673,
674,
675,
41,
677,
678,
679,
680,
49,
682,
683,
684,
50,
686,
687,
53,
689,
690,
691,
692,
693,
698,
699,
700,
55,
702,
59,
704,
705,
706,
707,
61,
709,
710,
711,
712,
713,
714,
65,
716,
717,
718,
719,
720,
721,
68,
723,
69,
70,
726,
727,
728,
729,
72,
731,
732,
733,
734,
735,
736,
737,
738,
739,
740,
741,
742,
743,
76,
745,
746
); QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Index Scan using sbom_components_pkey on sbom_components (cost=0.27..33.17 rows=84 width=39)
Index Cond: (id = ANY ('{29,660,661,37,38,664,665,666,667,668,39,670,40,672,673,674,675,41,677,678,679,680,49,682,683,684,50,686,687,53,689,690,691,692,693,698,699,700,55,702,59,704,705,706,707,61,709,710,711,712,713,714,65,716,717,718,719,720,721,68,723,69,70,726,727,728,729,72,731,732,733,734,735,736,737,738,739,740,741,742,743,76,745,746}'::bigint[]))
(2 rows)
Time: 2.085 msPreloading Sbom::ComponentVersion association
SELECT
sbom_component_versions.*
FROM
sbom_component_versions
WHERE
sbom_component_versions.id IN (
31,
686,
687,
688,
40,
690,
691,
692,
693,
694,
695,
41,
42,
698,
699,
43,
701,
702,
703,
704,
705,
706,
44,
708,
709,
710,
711,
53,
713,
714,
715,
54,
717,
718,
719,
57,
721,
722,
723,
724,
725,
726,
732,
733,
734,
735,
736,
63,
738,
739,
740,
741,
65,
743,
744,
745,
746,
747,
748,
749,
750,
751,
752,
753,
754,
755,
756,
757,
758,
759,
74,
75,
762,
763,
764,
765,
766,
77,
768,
769,
770,
771,
772,
773,
774,
775,
776,
777,
778,
779,
780,
781,
782,
783,
784,
785,
81,
787,
788,
789
); QUERY PLAN
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Index Scan using sbom_component_versions_pkey on sbom_component_versions (cost=0.28..37.99 rows=100 width=38)
Index Cond: (id = ANY ('{31,686,687,688,40,690,691,692,693,694,695,41,42,698,699,43,701,702,703,704,705,706,44,708,709,710,711,53,713,714,715,54,717,718,719,57,721,722,723,724,725,726,732,733,734,735,736,63,738,739,740,741,65,743,744,745,746,747,748,749,750,751,752,753,754,755,756,757,758,759,74,75,762,763,764,765,766,77,768,769,770,771,772,773,774,775,776,777,778,779,780,781,782,783,784,785,81,787,788,789}'::bigint[]))
(2 rows)
Time: 3.365 msPreloading Sbom::Source association (an occurrence may only have one source)
SELECT
sbom_sources.*
FROM
sbom_sources
WHERE
sbom_sources.id = 3; QUERY PLAN
---------------------------------------------------------------------------------------
Index Scan using sbom_sources_pkey on sbom_sources (cost=0.15..2.17 rows=1 width=58)
Index Cond: (id = 3)
(2 rows)
Time: 2.199 msMigrations
To support the new query on sbom_occurrences, this MR adds an index in [project_id, id].
Up
$ bin/rails db:migrate RAILS_ENV=test
main: == 20230215180605 IndexSbomOccurrencesOnProjectIdAndId: migrating =============
main: -- transaction_open?()
main: -> 0.0000s
main: -- view_exists?(:postgres_partitions)
main: -> 0.0047s
main: -- index_exists?(:sbom_occurrences, [:project_id, :id], {:name=>"index_sbom_occurrences_on_project_id_and_id", :algorithm=>:concurrently})
main: -> 0.0062s
main: -- execute("SET statement_timeout TO 0")
main: -> 0.0004s
main: -- add_index(:sbom_occurrences, [:project_id, :id], {:name=>"index_sbom_occurrences_on_project_id_and_id", :algorithm=>:concurrently})
main: -> 0.0049s
main: -- execute("RESET statement_timeout")
main: -> 0.0005s
main: == 20230215180605 IndexSbomOccurrencesOnProjectIdAndId: migrated (0.0428s) ====
ci: == 20230215180605 IndexSbomOccurrencesOnProjectIdAndId: migrating =============
ci: -- transaction_open?()
ci: -> 0.0000s
ci: -- view_exists?(:postgres_partitions)
ci: -> 0.0007s
ci: -- index_exists?(:sbom_occurrences, [:project_id, :id], {:name=>"index_sbom_occurrences_on_project_id_and_id", :algorithm=>:concurrently})
ci: -> 0.0060s
ci: -- execute("SET statement_timeout TO 0")
ci: -> 0.0004s
ci: -- add_index(:sbom_occurrences, [:project_id, :id], {:name=>"index_sbom_occurrences_on_project_id_and_id", :algorithm=>:concurrently})
ci: -> 0.0036s
ci: -- execute("RESET statement_timeout")
ci: -> 0.0005s
ci: == 20230215180605 IndexSbomOccurrencesOnProjectIdAndId: migrated (0.0307s) ====Down
$ bin/rails db:migrate:down:main VERSION=20230215180605 RAILS_ENV=test
main: == 20230215180605 IndexSbomOccurrencesOnProjectIdAndId: reverting =============
main: -- transaction_open?()
main: -> 0.0000s
main: -- view_exists?(:postgres_partitions)
main: -> 0.0044s
main: -- indexes(:sbom_occurrences)
main: -> 0.0076s
main: -- execute("SET statement_timeout TO 0")
main: -> 0.0006s
main: -- remove_index(:sbom_occurrences, {:algorithm=>:concurrently, :name=>"index_sbom_occurrences_on_project_id_and_id"})
main: -> 0.0038s
main: -- execute("RESET statement_timeout")
main: -> 0.0008s
main: == 20230215180605 IndexSbomOccurrencesOnProjectIdAndId: reverted (0.0389s) ====
$ bin/rails db:migrate:down:ci VERSION=20230215180605 RAILS_ENV=test
ci: == 20230215180605 IndexSbomOccurrencesOnProjectIdAndId: reverting =============
ci: -- transaction_open?()
ci: -> 0.0000s
ci: -- view_exists?(:postgres_partitions)
ci: -> 0.0992s
ci: -- indexes(:sbom_occurrences)
ci: -> 0.0043s
ci: -- execute("SET statement_timeout TO 0")
ci: -> 0.0005s
ci: -- remove_index(:sbom_occurrences, {:algorithm=>:concurrently, :name=>"index_sbom_occurrences_on_project_id_and_id"})
ci: -> 0.0017s
ci: -- execute("RESET statement_timeout")
ci: -> 0.0004s
ci: == 20230215180605 IndexSbomOccurrencesOnProjectIdAndId: reverted (0.1218s) ====Screenshots or screen recordings
Screenshots are required for UI changes, and strongly recommended for all other merge requests.
How to set up and validate locally
Numbered steps to set up and validate the change are strongly suggested.
-
Create a new project
-
Add the following
.gitlab-ci.ymlto the project:persist_sbom: image: alpine:latest script: - wget https://gitlab.com/-/snippets/2378046/raw/main/gl-sbom-go-go.cdx.json - wget https://gitlab.com/-/snippets/2378046/raw/main/gl-sbom-npm-npm.cdx.json artifacts: reports: cyclonedx: - gl-sbom-*.cdx.json -
Go to
<gdk_root>/-/graphql-explorer -
Issue the following graphql query:
query { project(fullPath: "your/project/path") { dependencies { nodes { name version packager location { blobPath path } } } } }
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Edited by Brian Williams