Connect to gemnasium-db repo

What does this MR do?

Make the analyzer leverage the gemnasium-db repo directly instead of connecting to the Gemnasium API.

This includes or will include:

• code refactoring !27 (merged)
• new advisory package, to manipulate gemnasium-db repo !28 (merged)
• version range resolvers !31 (merged)
  • native Ruby vrange CLI
  • vrange CLI based on gemnasium/semver
  • embed gemnasium-db repo
• re-organize vrange CLIs !41 (merged)
• native vrange CLIs
  • npm !42 (merged)
  • php !38 (merged)
  • python !36 (merged)
  • enable all native vrange CLIs !46 (merged)
  • add test cases based on gemnasium-db !47 (merged)
• use vrange in auto-remediation !34 (merged)
• provide an API gemnasium-maven and gemnasium-python can consume !37 (merged)
• update gemnasium-db at run-time !33 (merged)
• ensure local gemnasium-db repo is available at run-time !43 (merged)
• expose GEMNASIUM_DB_* variables for customization of the gemnasium-db repo !50 (merged)
• remove links to deps.sec.gitlab.com from reports (follow-up issue)

Warning! We may have to wait until the affected ranges are fixed in gemnasium-db before deploying this. See gitlab-org/security-products/gemnasium-db!121 (closed)

Commits will NOT be squashed to preserve refs to these MRs.

Future improvements

These are possible improvements that have been discussed during the review:

• back box testing
• using go-git
• better log when updating the local gemnasium-db repo

Related issues

gitlab-org/gitlab#14630 (closed)

Does this MR meet the acceptance criteria?

• Changelog entry added
• Documentation created/updated for GitLab EE; see gitlab-org/gitlab#14630 (closed)
• Documentation created/updated for this project, if necessary
• Documentation reviewed by technical writer or follow-up review issue created
• Tests added for this feature/bug
• Job definition updated, if necessary
• Conforms to the code review guidelines
• Conforms to the Go guidelines
• Security reports checked/validated by reviewer

changed milestone to %12.4

added Category:Dependency Scanning [DEPRECATED] backend backstage [DEPRECATED] devopssecure groupcomposition analysis labels

marked as a Work In Progress

changed the description

added 2 commits

• 14dde457 - Replace affected versions w/ affected range
• a0c84a6c - Stub Repo struct

Compare with previous version

added 2 commits

• 55834d70 - Refactor, reorganize packages
• 018ecfb7 - Merge branch '14630-pre-refactor' into '14630-use-gemnasium-db'

Compare with previous version

mentioned in issue gitlab-org/gitlab#14630 (closed)

added 2 commits

• 4c708c4a - Add advisory.Repo, Advisory
• 1c0e8194 - Merge branch '14630-advisory-pkg' into '14630-use-gemnasium-db'

Compare with previous version

changed the description

changed the description

added 2 commits

• 5c8cf2f5 - Emulate vrange.rb during tests
• a6163388 - Merge branch '14630-resolve-affected-range' into '14630-use-gemnasium-db'

Compare with previous version

marked the checklist item version range resolvers !31 (merged) as completed

changed the description

changed the description

changed the description

changed the description

added 2 commits

• f2343553 - Revert "ci"
• 952fa96d - Merge branch 'julian-python-version-checker' into '14630-use-gemnasium-db'

Compare with previous version

marked the checklist item python !36 (merged) as completed

added 1 commit

• c7b2de0c - Update changelog

Compare with previous version

mentioned in issue gitlab-org/gitlab#33341 (closed)

added 2 commits

• 638c17c3 - PHP version checker
• 1f63afba - Merge branch 'julian-php-version-checker' into '14630-use-gemnasium-db'

Compare with previous version

added 2 commits

• 24e22ee9 - Unit Test for Python Version Checker
• 7843d4a9 - Merge branch 'julian-python-version-checker-testcases' into '14630-use-gemnasium-db'

Compare with previous version

added 2 commits

• c5aa18c2 - Use vrange in yarn auto-remediate
• 6e0629bd - Merge branch '14630-vrange-in-auto-remediate' into '14630-use-gemnasium-db'

Compare with previous version

changed the description

mentioned in merge request !33 (merged)

changed the description

changed the description

mentioned in merge request !37 (merged)

changed the description

added 2 commits

• 7ccd77e9 - Check exit codes for PHP testcases
• d1f59a6b - Merge branch 'julian-fix-php-testcases' into '14630-use-gemnasium-db'

Compare with previous version

added 2 commits

• b277f789 - Vrange CLI for npm
• 7d5446dc - Merge branch '14630-vrange-npm' into '14630-use-gemnasium-db'

Compare with previous version

changed the description

marked the checklist item npm !42 (merged) as completed

added 2 commits

• ce6e4126 - Re-organize the vrange package
• 5c0661bc - Merge branch '14630-vrange-reorg' into '14630-use-gemnasium-db'

Compare with previous version

added 2 commits

• 76876135 - Add scanner.ScanFile, ScanReader
• 819cc8fa - Merge branch '14630-scan-file' into '14630-use-gemnasium-db'

Compare with previous version

marked the checklist item provide an API gemnasium-maven and gemnasium-python can consume !37 (merged) as completed

changed the description

marked the checklist item native vrange CLIs as completed

added 2 commits

• a4437555 - Update gemnasium db repo
• 42a37b9d - Merge branch '14630-update-gemnasium-db-repo' into '14630-use-gemnasium-db'

Compare with previous version

added 2 commits

• 22da17a2 - Remove package-lock.json
• 31297bda - Merge branch '14630-enable-vrange-cmds' into '14630-use-gemnasium-db'

Compare with previous version

marked the checklist item enable all native vrange CLIs !46 (merged) as completed

Failing QA job: PHP Composer

@adamcohen Currently we have extra advisories for the php-composer test project, but I don't know if that's legit. To be investigated on. See https://gitlab.com/gitlab-org/security-products/tests/php-composer/-/jobs/323686605

All other QA jobs are passing, see https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium/pipelines/89466184

cc @julianthome

Invalid affected ranges in gemnasium-db

@adamcohen gemnasium-db contains some YAML files where affected_range is invalid and does not comply with the version syntax corresponding to the package type. This has been spotted during QA and I've fixed CVE-2018-3728 to make it pass. This is no more than a workaround though, and @julianthome is currently running consistency checks using the vrange/* commands to make sure gemnasium-db contains valid ranges.

resolved all threads

Checking affected versions using Gemnasium API

We can leverage gemnasium-db-toolbox to list all the affected, fixed, and unaffected versions for a given advisory. I suggest we iterate over all the YAML files, retrieve these lists of versions, process them using the vrange/* CLIs (using the affected_range of the YAML file), and build a diff, to be checked manually. I'll work on that tomorrow. cc @adamcohen @julianthome

changed the description

marked the checklist item add test cases based on gemnasium-db !47 (merged) as incomplete

added 2 commits

• 316022ac - Fix, improve vrange CLIs, add test cases from gemnasium-db
• 2d72cba1 - Merge branch 'julian-test-cases' into '14630-use-gemnasium-db'

Compare with previous version

marked the checklist item add test cases based on gemnasium-db !47 (merged) as completed

mentioned in merge request gitlab-org/security-products/gemnasium-db-toolbox!10 (closed)

mentioned in merge request gitlab-org/security-products/gemnasium-db!121 (closed)

changed the description

changed the description

changed the description

mentioned in issue gl-retrospectives/secure#8 (closed)

added 2 commits

• 8c322fa1 - Ensure gemnasium-db has advisories
• dbcef04d - Merge branch '14630-check-gemnasium-db-repo' into '14630-use-gemnasium-db'

Compare with previous version

changed the description

unmarked as a Work In Progress

assigned to @adamcohen and unassigned @fcatteau

approved this merge request

assigned to @fcatteau and unassigned @adamcohen

added 1 commit

• b375393a - Use GEMNASIUM_REPO_* vars in Dockerfile

Compare with previous version

added 1 commit

• 4d4b0eec - Use GEMNASIUM_REPO_* vars in Dockerfile

Compare with previous version

changed the description

added 2 commits

• 50e19494 - Revert "add legacy version first and pass exception"
• 04c681e0 - Merge branch 'julian-python-legacy-version' into '14630-use-gemnasium-db'

Compare with previous version

mentioned in merge request !50 (merged)

added 2 commits

• 9b97e240 - Rename gemnasium-db variables
• 35d1a6c4 - Merge branch '14630-rename-gemnasium-db-vars' into '14630-use-gemnasium-db'

Compare with previous version

changed the description

resolved all threads

changed the description

I'm moving remove links to deps.sec.gitlab.com from reports out of this MR because A. this is not a blocker and B. it requires updating all the test projects, which takes time (mostly waiting for the pipelines) and would significantly delay this MR. cc @gonzoyumo

merged

mentioned in commit 6024edbf

mentioned in issue gitlab-org/gitlab#39404 (closed)

mentioned in issue gitlab-org/security-products/gemnasium-db#131 (closed)

mentioned in epic gitlab-org&1452 (closed)

mentioned in epic gitlab-org&1255 (closed)

mentioned in epic gitlab-org&1257 (closed)

mentioned in epic gitlab-org&1258 (closed)

mentioned in epic gitlab-org&1256 (closed)

mentioned in epic gitlab-org&1254 (closed)

mentioned in epic gitlab-org&1173 (closed)

mentioned in issue gitlab-org/gitlab#262400 (closed)

mentioned in issue gitlab-org/gitlab#327128 (closed)

changed the description

added Category:Software Composition Analysis SCA:Dependency Scanning labels