Parse dependency exports returned by multi-project Java builds (gemnasium-maven)

Summary

gemnasium-maven is currently unaware of multi-project Java builds, and multi-project builds are supported by doing the following:

1. find all supported build files, with no distinction b/w root projects and their sub-projects
2. iterate collected build files, pass each of them to the compatible builder, and make the builder exit right away if there's already a dependency export for the build file

This (mostly) works because:

• building the root project has the side-effect of building its children, and creating dependency exports for them
• the build file of the root project is processed before the build files of its children, because of the way the project finder walks the scanned directory

gemnasium-maven should be changed so that the dependency exports of a multi-project builds are listed when building the root project, instead of recursively finding build files and dependency exports matching these. This would be simpler, and more accurate. Also, it becomes possible to track the name of the sub-project where a dependency was detected.

Proposal

change gemnasiumm-maven so that:

• builders extract the output of the command that builds the root project, and return a list of all the dependency exports it creates
  • Gemnasium Maven plugin
  • Gemnasium Gradle plugin
  • sbt-dependency-graph; see gitlab-org/security-products/analyzers/gemnasium-maven!107 (merged)
• all the dependency exports returned by a multi-project build are scanned
• only the root project is found by the Finder

Implementation plan

• change builders in gemnasium-maven
  • change the Builder interface so that the Build function can return the dependency exports for the sub-projects, along with the dependency export for the root project gitlab-org/security-products/analyzers/gemnasium-maven!107 (merged)
  • update sbt builder: parse output of sbt and extract paths of DOT exports gitlab-org/security-products/analyzers/gemnasium-maven!107 (merged)
  • change the Analyze function so that additional exports returned by the builders are all processed gitlab-org/security-products/analyzers/gemnasium-maven!107 (merged)
  • update maven builder: parse output of Gemnasium Maven plugin and extract paths of JSON exports
  • update gradle builder: parse output of Gemnasium Gradle plugin and extract paths of JSON exports
  • change finder configuration and force the search mode to SearchSingleDir; the preset will be updated later on
• simplify finder; this will be done as part of #292952 (closed)
  • in gemnasium, remove SearchSingleTree mode and change gemnasium-maven's preset to SearchSingleDir
  • update gemnasium dependency in gemnasium-maven, and revert changes to the preset

NOTE: This introduces a discrepancy between the Builder defined in gemnasium-maven with the one defined in gemnasium and gemnasium-maven, but this will be addressed when moving the code of gemnasium-maven to gemnasium. See #292952 (closed)

Improvements

• code is simplified, in particular the finder
• gemnasium-maven and gemnasium-python are further aligned and both use the finder.SearchSingleDir search mode
• code is easier to debug; it no longer inspects directories that are not part of the multi-project build

Risks

• The location of vulnerabilities detected in sub-projects might change, resulting in the loss of vulnerability feedback.
• The dependency_files[].path for the sub-projects might change, in the report. This would only change the Dependency List page though.
• Dependencies of the sub-projects might merged into the dependencies of the root project, the dependency_files field of the report.
• In case their are multiple root projects in a repo, the new implementation might pick a different one. That said, there are no specs to establish what root project the analyzer should pick and scan.

Involved components

• https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium
• https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven

Optional: Intended side effects

Sub-projects can be scanned as long as they have a dependency file export, even if there's no build file is detected in the directory of the sub-project. This fixes a bug where Gradle sub-projects are skipped if the build file name isn't build.gradle. See #322346 (closed)

Optional: Missing test coverage

/cc @ifrenkel @gonzoyumo @adamcohen