Add `purl_type` to SBoM components
Why are we doing this work
Currently, we determine the uniqueness of an SBoM component using its name and the CycloneDX type. As explained in this comment, this is problematic because we may have collisions between packages which have the same name but exist in different package indices (ex: PyPI and Rubygems).
The
component_typeis the type as CycloneDX defines it, meaning that all programming language libraries will have thelibrarytype. PyPI has a package namedpgand Rubygems also has a package namedpg. These will both end up having{ component_type: :library, component_name: "pg" }and will collide, even though they are different dependencies.
As a solution to this, we should add a column onto sbom_components to store the
package URL type.
We will then change the uniqueness constraint for sbom_components to be unique by component_type + purl_type + name.
Relevant links
Non-functional requirements
-
Documentation: -
Feature flag: -
Performance: -
Testing:
Implementation plan
- Add a
purl_typecolumn ontosbom_components. This should be an enum (smallint) and it should have enums for each of the package url types which are currently supported by dependency scanning. - Remove
index_sbom_components_on_component_type_and_name - Add a new index on
[:component_type, :purl_type, :name] - Add a package-url parser to GitLab. Perhaps https://github.com/package-url/packageurl-ruby.
- Update
lib/gitlab/ci/parsers/sbom/cyclonedx.rbto parse thepurlfrom the component and add thepurl_typeto the component object. - Update the
IngestComponentstask to store thepurl_typeand be unique by[:component_type, :purl_type, :name]
Verification steps
-
Enable the feature flag on your project (please comment / tick this in the feature flag rollout issue)
-
Create a new project from a template, use the NodeJS/Express template. Make sure that the group which you create this project under has an ultimate plan.
-
Create a
.gitlab-ci.ymlfile with this configuration:include: - template: Security/Dependency-Scanning.gitlab-ci.yml -
Verify that the
gemnasium-dependency_scanningoutputs agl-sbom-npm-npm.cdx.jsonartifact -
Use teleport to connect to the db console. For production, this will require an access request, but this will be automatically sent to the
#infrastructure-loungechannel when you runtsh loginand is usually approved very quickly. -
Take this query an paste it inside an editor. Replace
YOUR_PIPELINE_IDwith the ID of your CI pipeline which produced thegl-sbom-npm-npm.cdx.jsonartifact.select name, version, component_type, purl_type, source_id from sbom_components inner join sbom_component_versions on sbom_components.id = sbom_component_versions.component_id inner join sbom_occurrences on sbom_component_versions.id = sbom_occurrences.component_version_id where pipeline_id = YOUR_PIPELINE_ID; -
Paste the query with your pipeline ID into the database console. Verify the
purl_typeis present. See https://gitlab.com/gitlab-org/gitlab/-/blob/63a47ad481690a6ecdd9d2f8371ab2c22d2e3fda/app/models/concerns/enums/sbom.rb#L9 for the enum mapping.
Example data
name | version | component_type | purl_type | source_id
----------------------+--------------+----------------+-----------+-----------
@types/babel-types | 7.0.4 | 0 | 6 |
@types/babylon | 6.16.3 | 0 | 6 |
accepts | 1.3.5 | 0 | 6 |
acorn | 3.3.0 | 0 | 6 |
acorn | 4.0.13 | 0 | 6 |
acorn-globals | 3.1.0 | 0 | 6 |
align-text | 0.1.4 | 0 | 6 |
amdefine | 1.0.1 | 0 | 6 |
array-flatten | 1.1.1 | 0 | 6 |
asap | 2.0.6 | 0 | 6 |
asynckit | 0.4.0 | 0 | 6 |
babel-runtime | 6.26.0 | 0 | 6 |
babel-types | 6.26.0 | 0 | 6 |
babylon | 6.18.0 | 0 | 6 |
balanced-match | 1.0.0 | 0 | 6 |
basic-auth | 2.0.0 | 0 | 6 |
body-parser | 1.18.2 | 0 | 6 |
brace-expansion | 1.1.11 | 0 | 6 |
browser-stdout | 1.3.1 | 0 | 6 |
bytes | 3.0.0 | 0 | 6 |
camelcase | 1.2.1 | 0 | 6 |
center-align | 0.1.3 | 0 | 6 |
character-parser | 2.2.0 | 0 | 6 |
clean-css | 3.4.28 | 0 | 6 |
cliui | 2.1.0 | 0 | 6 |
combined-stream | 1.0.6 | 0 | 6 |
commander | 2.15.1 | 0 | 6 |
commander | 2.8.1 | 0 | 6 |
component-emitter | 1.2.1 | 0 | 6 |
concat-map | 0.0.1 | 0 | 6 |
constantinople | 3.1.2 | 0 | 6 |
content-disposition | 0.5.2 | 0 | 6 |
content-type | 1.0.4 | 0 | 6 |
cookie | 0.3.1 | 0 | 6 |
cookie-parser | 1.4.3 | 0 | 6 |
cookie-signature | 1.0.6 | 0 | 6 |
cookiejar | 2.1.2 | 0 | 6 |
core-js | 2.5.7 | 0 | 6 |
core-util-is | 1.0.2 | 0 | 6 |
debug | 2.6.9 | 0 | 6 |
debug | 3.1.0 | 0 | 6 |
decamelize | 1.2.0 | 0 | 6 |
delayed-stream | 1.0.0 | 0 | 6 |
depd | 1.1.1 | 0 | 6 |
depd | 1.1.2 | 0 | 6 |
destroy | 1.0.4 | 0 | 6 |
diff | 3.5.0 | 0 | 6 |
doctypes | 1.1.0 | 0 | 6 |
ee-first | 1.1.1 | 0 | 6 |
encodeurl | 1.0.2 | 0 | 6 |
escape-html | 1.0.3 | 0 | 6 |
escape-string-regexp | 1.0.5 | 0 | 6 |
esutils | 2.0.2 | 0 | 6 |
etag | 1.8.1 | 0 | 6 |
express | 4.16.3 | 0 | 6 |
extend | 3.0.2 | 0 | 6 |
finalhandler | 1.1.1 | 0 | 6 |
form-data | 2.3.2 | 0 | 6 |
formidable | 1.2.1 | 0 | 6 |
forwarded | 0.1.2 | 0 | 6 |
fresh | 0.5.2 | 0 | 6 |
fs.realpath | 1.0.0 | 0 | 6 |
function-bind | 1.1.1 | 0 | 6 |
glob | 7.1.2 | 0 | 6 |
graceful-readlink | 1.0.1 | 0 | 6 |
growl | 1.10.5 | 0 | 6 |
has | 1.0.3 | 0 | 6 |
has-flag | 3.0.0 | 0 | 6 |
he | 1.1.1 | 0 | 6 |
http-errors | 1.6.2 | 0 | 6 |
http-errors | 1.6.3 | 0 | 6 |
iconv-lite | 0.4.19 | 0 | 6 |
inflight | 1.0.6 | 0 | 6 |
inherits | 2.0.3 | 0 | 6 |
ipaddr.js | 1.8.0 | 0 | 6 |
is-buffer | 1.1.6 | 0 | 6 |
is-expression | 3.0.0 | 0 | 6 |
is-promise | 2.1.0 | 0 | 6 |
is-regex | 1.0.4 | 0 | 6 |
isarray | 1.0.0 | 0 | 6 |
js-stringify | 1.0.2 | 0 | 6 |
jstransformer | 1.0.0 | 0 | 6 |
kind-of | 3.2.2 | 0 | 6 |
lazy-cache | 1.0.4 | 0 | 6 |
lodash | 4.17.10 | 0 | 6 |
longest | 1.0.1 | 0 | 6 |
media-typer | 0.3.0 | 0 | 6 |
merge-descriptors | 1.0.1 | 0 | 6 |
methods | 1.1.2 | 0 | 6 |
mime | 1.4.1 | 0 | 6 |
mime-db | 1.35.0 | 0 | 6 |
mime-types | 2.1.19 | 0 | 6 |
minimatch | 3.0.4 | 0 | 6 |
minimist | 0.0.8 | 0 | 6 |
mkdirp | 0.5.1 | 0 | 6 |
mocha | 5.2.0 | 0 | 6 |
morgan | 1.9.0 | 0 | 6 |
ms | 2.0.0 | 0 | 6 |
negotiator | 0.6.1 | 0 | 6 |
object-assign | 4.1.1 | 0 | 6 |
on-finished | 2.3.0 | 0 | 6 |
on-headers | 1.0.1 | 0 | 6 |
once | 1.4.0 | 0 | 6 |
parseurl | 1.3.2 | 0 | 6 |
path-is-absolute | 1.0.1 | 0 | 6 |
path-parse | 1.0.5 | 0 | 6 |
path-to-regexp | 0.1.7 | 0 | 6 |
process-nextick-args | 2.0.0 | 0 | 6 |
promise | 7.3.1 | 0 | 6 |
proxy-addr | 2.0.4 | 0 | 6 |
pug | 2.0.0-beta11 | 0 | 6 |
pug-attrs | 2.0.3 | 0 | 6 |
pug-code-gen | 1.1.1 | 0 | 6 |
pug-error | 1.3.2 | 0 | 6 |
pug-filters | 2.1.5 | 0 | 6 |
pug-lexer | 3.1.0 | 0 | 6 |
pug-linker | 2.0.3 | 0 | 6 |
pug-load | 2.0.11 | 0 | 6 |
pug-parser | 2.0.2 | 0 | 6 |
pug-runtime | 2.0.4 | 0 | 6 |
pug-strip-comments | 1.0.3 | 0 | 6 |
pug-walk | 1.1.7 | 0 | 6 |
qs | 6.5.1 | 0 | 6 |
range-parser | 1.2.0 | 0 | 6 |
raw-body | 2.3.2 | 0 | 6 |
readable-stream | 2.3.6 | 0 | 6 |
regenerator-runtime | 0.11.1 | 0 | 6 |
repeat-string | 1.6.1 | 0 | 6 |
resolve | 1.8.1 | 0 | 6 |
right-align | 0.1.3 | 0 | 6 |
safe-buffer | 5.1.1 | 0 | 6 |
send | 0.16.2 | 0 | 6 |
serve-static | 1.13.2 | 0 | 6 |
setprototypeof | 1.0.3 | 0 | 6 |
setprototypeof | 1.1.0 | 0 | 6 |
source-map | 0.4.4 | 0 | 6 |
source-map | 0.5.7 | 0 | 6 |
statuses | 1.4.0 | 0 | 6 |
string_decoder | 1.1.1 | 0 | 6 |
superagent | 3.8.2 | 0 | 6 |
supertest | 3.1.0 | 0 | 6 |
supports-color | 5.4.0 | 0 | 6 |
to-fast-properties | 1.0.3 | 0 | 6 |
token-stream | 0.0.1 | 0 | 6 |
type-is | 1.6.16 | 0 | 6 |
uglify-js | 2.8.29 | 0 | 6 |
uglify-to-browserify | 1.0.2 | 0 | 6 |
unpipe | 1.0.0 | 0 | 6 |
util-deprecate | 1.0.2 | 0 | 6 |
utils-merge | 1.0.1 | 0 | 6 |
vary | 1.1.2 | 0 | 6 |
void-elements | 2.0.1 | 0 | 6 |
window-size | 0.1.0 | 0 | 6 |
with | 5.1.1 | 0 | 6 |
wordwrap | 0.0.2 | 0 | 6 |
wrappy | 1.0.2 | 0 | 6 |
yargs | 3.10.0 | 0 | 6 |
(157 rows)