WIP: Add vrange JSON files from server & client

What does this MR do?

Add vrange-compatible JSON files based on what's returned by the Gemnasium API, and process them using vrange CLIs. These correspond to file extension .vrange.server.json and .vrange.client.json, respectively.

This is used to compare the affected and unaffected versions with the ones found by the new Gemnasium implementation using gemnasium-db. See gitlab-org/security-products/analyzers/gemnasium!25 (merged)

WARNING! DO NOT MERGE!

Related issues

gitlab-org/gitlab#14630 (closed)

/cc @julianthome

changed milestone to %12.5

added Category:Dependency Scanning [DEPRECATED] backend backstage [DEPRECATED] devopssecure groupcomposition analysis labels

@julianthome I've used json-diff to build vrange_diff_files, the list of the 298 YAML of files for which there's a difference, out of 1485.

Here are two random things I've spotted:

• On the server, pre-releases of pypi/aubio/CVE-2018-19800.yml are included. The server is right, though the affected range is not right, in my opinion.
• On the client, pre-releases of gem/yard/CVE-2017-17042.yml are included. The client is right.

There's a lot to review. Could you help me with that?

I probably need a tool to better present the differences between the JSON files. Working on that. cc @julianthome

mentioned in merge request gitlab-org/security-products/analyzers/gemnasium!25 (merged)

Just to make sure that I understand the test correctly; this test is for checking whether the data returned by the gemnasium-API (and passed through the vrange tools) is consistent with what we get from the vrange tools when executed on the data from gemnasium-db? So you generated JSON files (compatible with the vrange protocol) that contain the packages and affected ranges returned by the gemnasium-API as well as gemnasium-db and differ the outputs of the vrange tools against one another?! So are these inconsistencies caused by syncing problems (e.g., if advisories from gemnasium-db haven't been synced) or can there also be other reasons for inconsistencies?

On the server, pre-releases of pypi/aubio/CVE-2018-19800.yml are included. The server is right, though the affected range is not right, in my opinion.

Based on https://nvd.nist.gov/vuln/detail/CVE-2018-19800, the affected range seems to be okay I suppose, doesn't it?

There's a lot to review. Could you help me with that?

Yes, sure.

added 1 commit

• d2760428 - Add command to compare vrange JSON files

Compare with previous version

mentioned in issue gitlab-org/gitlab#14630 (closed)

mentioned in merge request gitlab-org/security-products/analyzers/gemnasium!48 (closed)

gem

I've reviewed the diff for the Ruby gems and everything is OK. The new Gemnasium is a real improvement here, adding many affected versions that were previously ignored by the Gemnasium API.

vrange_diff_gem

npm

The diff for the npm packages is OK (adding affected pre-releases that were previously ignored), except for two cases where we may update the affected range to catch the pre-releases:

npm/galenframework-cli/GMS-2016-90.yml '<2.3.1' +2.3.1-3 +2.3.1-2 +2.3.1-5 +2.3.1-6 +2.3.0
npm/go-ipfs-dep/GMS-2016-96.yml '<0.4.4' +0.4.4-1

Nothing wrong with range checking.

See vrange_diff_npm

Maven

Diff for Maven packages is OK except for 4 lines. These 3 changes seem to be plain wrong:

maven/org.picketlink/picketlink-common/cvE-2014-3530.yml '(,2.6.0.Final],[3-alpha0,3.0.0-2013Feb20]' -3.0.0.Beta2 -3.0.0.Alpha1 -3.0.0.Beta1 +2.5.5.SP11 +2.5.4.SP17 +2.5.5.SP9 +2.5.5.SP12 +2.5.4.SP18 +2.5.5.SP10
maven/org.picketlink/picketlink-federation/CVE-2014-7827.yml '[2-alpha0,2.5.3.SP13],[2.6-alpha0,2.6.1],[2.7-alpha0,2.7.1.Beta2]' +2.7.0.CR1-20140924 +2.7.0.Beta1-20140731
maven/org.eclipse.jetty/jetty-http/CVE-2015-2080.yml '[9.2.3,9.3-alpha0),[9.2-alpha0,9.2.8]' +9.2.26.v20180806 +9.2.27.v20190403 +9.2.25.v20180606 +9.2.24.v20180105 +9.2.28.v20190418 +9.2.23.v20171218

It looks like there's an issue with range checking, and that some pre-releases get excluded when they should not. This is surprising though, since the CLI tool used to perform the check and the Gemnasium API both rely on the gemnasium/semver Go library.

We need to investigate on this one:

maven/org.sonatype.nexus.plugins/nexus-yum-repository-plugin/CVE-2019-5475.yml '[2.0,2.14.13]' -2.14.13-01

I don't remember if 2.14.13-01 is a pre-release or a post-release in the case of Maven.

See vrange_diff_maven

Packagist

Diff for Packagist packages is OK except for 31 advisories where the affected range is ambiguous, and it's not clear whether pre-releases should be included or not (when they match a lower or upper boundary). There's nothing wrong with range checking though.

See vrange_diff_packagist

Pypi

There are 8 diffs that are clearly not OK, out of 39. For instance >5.0a,<5.0rc2||<4.3.7 should include 5.0b1.post1, but it got removed.

NO pypi/Products.CMFPlone/CVE-2015-7316.yml '>5.0a,<5.0rc2||<4.3.7' -5.0b1.post1
NO pypi/buildbot/CVE-2019-7313.yml '<1.8.1' -0.7.11p2 -0.8.6p1 -0.8.7p1 -0.7.10p1 -0.8.4p1 -0.8.3p1 -0.8.4p2 -0.8.1p1 -0.7.11p1 -0.7.11p3
NO pypi/buildbot/CVE-2019-12300.yml '<1.8.2||>=2.0.0,<2.3.1' -0.7.10p1 -0.7.11p3 -0.8.3p1 -0.8.4p2 -0.8.6p1 -0.8.4p1 -0.8.7p1 -0.7.11p2 -0.8.1p1 -0.7.11p1
NO pypi/Django/CVE-2017-7234.yml '>=1.11alpha,<1.11||>=1.10.0alpha,<1.10.7||>=1.9.0alpha,<1.9.13||<1.8.18' -1.11b1 -1.11a1 -1.11rc1 +1.7a2
NO pypi/Django/CVE-2017-7233.yml '>=1.11alpha,<1.11||>=1.10.0alpha,<1.10.7||>=1.9.0alpha,<1.9.13||<1.8.18' -1.11rc1 -1.11a1 -1.11b1 +1.7a2
NO pypi/Django/CVE-2019-14232.yml '>=1.11.0 <1.11.23 || >=2.1.0 <2.1.11 || >=2.2.0 <2.2.4' +2.0.5 +2.2.4 +2.0.2 +2.0a1 +2.0.9 +2.2 +2.0.10 +2.0.7 +2.0.4 +2.0.3 +2.0.13 +2.0b1 +2.1 +1.11.23 +2.0rc1 +1.11 +2.0.1 +2.0 +2.1.11 +2.0.12 +2.0.8 +2.0.6
NO pypi/Django/CVE-2019-14233.yml '>=1.11.0 <1.11.23 || >=2.1.0 <2.1.11 || >=2.2.0 <2.2.4' +2.2 +2.0.4 +2.0.13 +2.0rc1 +2.0.10 +2.0.8 +2.2.4 +2.0.12 +2.1.11 +2.0a1 +2.1 +2.0.2 +2.0.5 +2.0b1 +2.0.6 +2.0.9 +2.0.7 +2.0 +1.11.23 +2.0.1 +1.11 +2.0.3
NO pypi/Django/CVE-2019-14235.yml '>=1.11.0 <1.11.23 || >=2.1.0 <2.1.11 || >=2.2.0 <2.2.4' -1.8.3 -1.6.11 -1.7.6 -1.7.5 -1.6.10 -1.6.5 -1.3.7 -1.3.1 -1.9.10 -1.9.5 -1.4.22 -1.5.8 -1.4.10 -1.8.17 -1.8.13 -1.8.8 -1.5.12 -1.4.13 -1.9rc2 -1.4.8 -1.3.4 -1.2.6 -1.4.3 -1.10.8 -1.10.6 -1.10b1 -1.8 -1.4.4 -1.1.4 -1.0.1 -1.3.3 -1.10.7 -1.7.7 -1.4.19 -1.6.6 -1.5.5 -1.9.12 -1.8.12 -1.8b2 -1.6.8 -1.2.3 -1.8.18 -1.10.1 -1.3.2 -1.8.15 -1.7.11 -1.8.4 -1.7.3 -1.2.4 -1.9b1 -1.4.18 -1.4.2 -1.7.10 -1.7 -1.3.5 -1.3 -1.1.3 -1.4.14 -1.8.6 -1.8c1 -1.7.4 -1.7.2 -1.5.7 -1.6.2 -1.6.1 -1.5.1 -1.3.6 -1.5.11 -1.5.4 -1.4.1 -1.0.3 -1.10 -1.1.1 -1.7a2 -1.10.4 -1.7.8 -1.4.20 -1.6.9 -1.6.3 -1.4.7 -1.4.5 -1.0.2 -1.1 -1.8.5 -1.4.21 -1.2 -1.6.4 -1.9 -1.9.8 -1.9.7 -1.8.11 -1.4.16 -1.5.9 -1.5.2 -1.5 -1.10rc1 -1.8a1 -1.7.1 -1.2.2 -1.5.10 -1.5.3 -1.9.4 -1.10.2 -1.9.6 -1.8.9 -1.4.11 -1.9.1 -1.7.9 -1.5.6 -1.4 -1.0.4 -1.8.16 -1.9.9 -1.8.1 -1.4.9 -1.8.19 -1.8.10 -1.9.2 -1.8.7 -1.8.2 -1.4.17 -1.9rc1 -1.9.13 -1.10.5 -1.10.3 -1.4.15 -1.6 -1.2.1 -1.1.2 -1.8.14 -1.10a1 -1.9a1 -1.4.12 -1.4.6 -1.9.11 -1.8b1 -1.6.7 -1.2.7 -1.9.3 -1.2.5 +2.1.11 +1.11.23 +2.2.4

Also, it looks like 0.2.1 should be rewritten as ==0.2.1, though I don't if this is a bug in the parser itself.

NO pypi/django-crm/CVE-2019-11457.yml '0.2.1' -0.2.1

Finally, there are cases 6 cases where the affected range is ambiguous, and this should be addressed by updating the YAML files.

?? pypi/aubio/CVE-2018-19802.yml '>=0.4.0,<0.4.9' -0.4.0alpha1
?? pypi/aubio/CVE-2018-19801.yml '>=0.4.0,<0.4.9' -0.4.0alpha1
?? pypi/aubio/CVE-2018-19800.yml '>=0.4.0,<0.4.9' -0.4.0alpha1
?? pypi/apache-airflow/CVE-2018-20244.yml '<1.10.2' -1.10.2rc3 -1.10.2rc1 -1.10.2b2 -1.10.2rc2
?? pypi/apache-airflow/CVE-2018-20245.yml '<1.10.1' -1.10.1rc2 -1.10.1b1
?? pypi/salt/CVE-2014-3563.yml '>=2014.1.0,<2014.1.10' -2014.1.0rc1 -2014.1.0rc3 -2014.1.0rc2 +2014.1.3 +2014.1.2 +2014.1.8 +2014.1.7 +2014.1.6 +2014.1.5 +2014.1.4 +2014.1.9

See vrange_diff_pypi

@julianthome I'm done reviewing the diff. Could you please take a look, and comment on this review?

assigned to @julianthome and unassigned @fcatteau

mentioned in merge request !123 (merged)

mentioned in commit 84ea828b

mentioned in merge request !124 (merged)

mentioned in commit 5a3549bb

mentioned in merge request !125 (merged)

mentioned in merge request gitlab-org/security-products/analyzers/gemnasium!49 (merged)

mentioned in commit f7f6e48c

resolved all threads

assigned to @fcatteau

unassigned @julianthome

mentioned in commit e4881a04

resolved all threads

We've got everything covered in these MRs:

• !123 (merged)
• !124 (merged)
• !125 (merged)
• !126 (merged)
• gitlab-org/security-products/analyzers/gemnasium!49 (merged)

Closing.

closed

mentioned in merge request gemnasium-db-toolbox!10 (closed)

added Category:Software Composition Analysis SCA:Dependency Scanning labels