Preview using gemnasium/vrange, package registries (no packagist support)

What does this MR do?

Use gemnasium-db as a single source of truth instead of connecting to the Gemnasium API.

Preview is implemented by comparing the affected range of the advisory with the versions listed on the package registries.

TODO

• Rewrite Git history before merging

PHP's packagist.org is out of scope.

See gitlab-org/gitlab#11849 (closed)

Changes

• remove client for the Gemnasium API
• add clients to connect to the package registries (rubygems.org, npmjs.com, etc) and list package versions with their publication dates; this code comes from the Gemnasium Server (package-syncer service)
• build Docker image on top of gemnasium:2.3.0, top leverage its vrange library to evaluate the affected of a security advisory
• remove univers sub-package since it was a subset of want vrange covers
• remove prefill and filter sub-commands from the CLI; they are no longer needed
• rewrite the preview sub-command using package registry clients and the vrange library

Warning! The new client shares no code with the old one, even though it lives in the same directory. There's nothing to compare.

Usage

Preview a security advisory using the Docker image and a clone of gemnasium-db:

$ docker run -ti --rm  --volume $PWD:/gemnasium-db registry.gitlab.com/gitlab-org/security-products/gemnasium-db-toolbox:11849-gemnasium-db preview pypi/cryptography/CVE-2016-9243.yml 

Preview using local binary:

% ./gemnasium-db-toolbox preview --vrange-dir ../analyzers/gemnasium/vrange ../gemnasium-db/npm/shout/GMS-2015-2.yml|head -n 22
  0.53.0 2016-01-07T06:42:31.869Z
  0.52.0 2015-10-19T04:31:19.094Z
  0.51.2 2015-09-20T16:57:21.808Z
  0.51.1 2015-04-29T19:56:52.668Z
  0.51.0 2015-04-16T08:21:47.919Z
+ 0.50.0 2015-01-22T17:54:02.785Z
! 0.49.3 2015-01-11T23:00:03.500Z
! 0.49.2 2015-01-04T03:01:13.348Z
! 0.49.1 2015-01-04T02:52:46.391Z
! 0.49.0 2014-12-23T12:27:37.377Z
! 0.48.0 2014-12-12T00:03:51.153Z
! 0.47.0 2014-11-18T23:01:27.231Z
! 0.46.0 2014-11-13T23:25:32.403Z
! 0.45.5 2014-11-06T12:02:04.271Z
! 0.45.4 2014-11-05T22:22:31.755Z
! 0.45.3 2014-10-27T22:02:07.629Z
! 0.45.2 2014-10-16T21:36:10.925Z
! 0.45.1 2014-10-14T22:25:53.028Z
! 0.45.0 2014-10-14T21:41:59.364Z
! 0.44.0 2014-10-10T23:14:24.748Z
  0.43.2 2014-10-10T17:50:44.728Z
  0.43.1 2014-10-09T14:37:55.632Z

Check multiple advisories using a local binary:

% ./gemnasium-db-toolbox check $(find gemnasium-db -name '*.yml') && echo OK
gemnasium-db/pypi/Cobbler/CVE-2014-3225.yml yaml: line 1: did not find expected key
gemnasium-db/maven/com.fasterxml.jackson.core/jackson-databind/CVE-2019-16335.yml yaml: line 6: did not find expected key
2019/11/05 09:04:35 validation error

% ./gemnasium-db-toolbox check $(find gemnasium-db -name '*.yml') && echo OK
OK

Related issues

gitlab-org/gitlab#11849 (closed)