Assess accuracy of semver_dialects using gemnasium/vrange
Problem to solve
We'd like to use semver_dialects
to implement Continuous Scanning in the backend,
and to support all the languages and package managers Gemnasium (Dependency Scanning) supports today.
However, semver_dialects
might not be as accurate as Gemnasium's vrange library,
and this might result in false positives and false negatives.
See #363073 (comment 970572115)
Proposal
Leverage gemnasium-db to compare semver_dialects
to vrange
, report discrepancies, and create follow-up issues to address them.
For each YAML file of gemnasium-db
, do the following:
- Query the package registry to list package versions.
- Find affected, fixed, and non-affected versions using
gemnasium/vrange
. - Find affected, fixed, and non-affected versions using
semver_dialects
. - Report inconsistencies.
We can either query the API of the package registry or use the package manager to list the versions of a package.
To find the affected and non-affected versions, go through the versions listed on the package registry, and evaluate the affected_range
of the YAML file.
Also, find package versions that match the fixed_versions
of the YAML file.
See #220286 (comment 1044076127)
Implementation plan
The proposal above is likely the best way to do an assessment of the semver_dialects
gem. The solution will also allow our team to facilitate Ensure consistency b/w semver_dialects and vran... (#369239).
-
add vrange command to gemnasium allowing queries given a range and a version -
create a project for running this test - ruby optimal here since
semver_dialects
is a ruby gem and this matching capability will be used in the rails monolith
- ruby optimal here since
-
check out advisory db -
iterate over each package manager directory (e.g. pypi
,gem
)- iterate over each directory corresponding to a package
- get list of official versions for package (to simplify configuration try to use docker as much as possible; this can be used in Ensure consistency b/w semver_dialects and vran... (#369239) under docker-dind image)
- iterate over each advisory
- check
version
of package againstadvisory.affected_range
- using
vrange
- using
semver_dialects
- using
-
capture matches, mismatches and which range functionality they belong to using a greppable format - capture to db that can be saved to disk and queries via
jq
(for example)
- capture to db that can be saved to disk and queries via
- check
- iterate over each directory corresponding to a package
-
commit code used to generate this check to make it repeatable for new semver types - https://gitlab.com/gitlab-org/security-products/tests/semver-assessment -
report results - #369238 (comment 1114975584) -
create follow-up issues
/cc @julianthome @bwill