Draft: Test version parsing against export v1 of licenses

What does this MR do?

Test version parsing against export v1 of license data.

This is built on top of Draft: Test parsing and comparison against expo... (!85).

To speed things up and to produce a readable output, it only count what's correct, and reports the percentage of correct results if not 100%.

See Integration tests for semver_dialects using lic... (gitlab-org/gitlab#462854 - closed)

Usage

First you need to copy license data from the GCP bucket locally. You only need the v1 directory, and can skip v2. You can follow the instructions for offline environments.

The environment variable LICENSES_BUCKET_DIR must contain the path of the local copy.

PKG_METADATA_BUCKET="prod-export-license-bucket-1a6c642fc4de57d4"
LICENSES_BUCKET_DIR="$HOME/license-db/$PKG_METADATA_BUCKET/"

gsutil -o "GSUtil:parallel_process_count=1" -m rsync -r -d -y "^v2\/" "gs://$PKG_METADATA_BUCKET" "$LICENSES_BUCKET_DIR"

The following line runs the specs using all the CSV files found LICENSES_BUCKET_DIR and all supported package types.

bundle exec rspec spec/license_scanning_spec.rb

You can limit the checks to N lines/JSON objects per package type using LINES_PER_PKG_TYPE.

LINES_PER_PKG_TYPE=10000 bundle exec rspec spec/license_scanning_spec.rb

You can use LICENSES_PKG_TYPES to only check specific package types. These types correspond to sub-directories of v1, not to the ones used by the gem and by gemnasium-db.

LICENSES_PKG_TYPES="conan rubygem" bundle exec rspec spec/license_scanning_spec.rb -f p