Draft: Test parsing and comparison against export v2 of licenses

What does this MR do?

Test the gem against export v2 of license data.

• Check that the lowest and highest versions can be parsed.
• Check that the highest version is greater than or equal to the lowest version.

To speed things up and to produce a readable output, it only count what's correct, and reports the percentage of correct results if not 100%.

.F.FFF.F.F.FFF.F

Failures:

  1) SemverDialects with licenses export v2 for conan package compares boundaries as expected
     Failure/Error: expect(100.0*valid_comparisons_count/comparisons_count).to eq(100.0)

       expected: 100.0
            got: 99.95674116798847

       (compared using ==)
     # ./spec/license_scanning_spec.rb:79:in `block (5 levels) in <top (required)>'

  2) SemverDialects with licenses export v2 for go package compares boundaries as expected
     Failure/Error: expect(100.0*valid_comparisons_count/comparisons_count).to eq(100.0)

       expected: 100.0
            got: 99.99870955869152

       (compared using ==)
     # ./spec/license_scanning_spec.rb:79:in `block (5 levels) in <top (required)>'

See Integration tests for semver_dialects using lic... (gitlab-org/gitlab#462854 - closed)

Usage

First you need to copy license data from the GCP bucket locally. You only need the v2 directory, and can skip v1. You can follow the instructions for offline environments.

The environment variable LICENSES_BUCKET_DIR must contain the path of the local copy.

PKG_METADATA_BUCKET="prod-export-license-bucket-1a6c642fc4de57d4"
LICENSES_BUCKET_DIR="$HOME/license-db/$PKG_METADATA_BUCKET/"

gsutil -o "GSUtil:parallel_process_count=1" -m rsync -r -d -y "^v1\/" "gs://$PKG_METADATA_BUCKET" "$LICENSES_BUCKET_DIR"

The following line runs the specs using all the NDJSON files found LICENSES_BUCKET_DIR and all supported package types.

bundle exec rspec spec/license_scanning_spec.rb

You can limit the checks to N lines/JSON objects per package type using LINES_PER_PKG_TYPE.

LINES_PER_PKG_TYPE=10000 bundle exec rspec spec/license_scanning_spec.rb

You can use LICENSES_PKG_TYPES to only check specific package types. These types correspond to sub-directories of v2, not to the ones used by the gem and by gemnasium-db.

LICENSES_PKG_TYPES="conan rubygem" bundle exec rspec spec/license_scanning_spec.rb -f p