Parse dependency exports returned by multi-project Java builds (gemnasium-maven)
Summary
gemnasium-maven
is currently unaware of multi-project Java builds, and multi-project builds are supported by doing the following:
- find all supported build files, with no distinction b/w root projects and their sub-projects
- iterate collected build files, pass each of them to the compatible builder, and make the builder exit right away if there's already a dependency export for the build file
This (mostly) works because:
- building the root project has the side-effect of building its children, and creating dependency exports for them
- the build file of the root project is processed before the build files of its children, because of the way the project finder walks the scanned directory
gemnasium-maven
should be changed so that the dependency exports of a multi-project builds are listed when building the root project, instead of recursively finding build files and dependency exports matching these. This would be simpler, and more accurate. Also, it becomes possible to track the name of the sub-project where a dependency was detected.
Proposal
change gemnasiumm-maven
so that:
- builders extract the output of the command that builds the root project, and return a list of all the dependency exports it creates
- Gemnasium Maven plugin
- Gemnasium Gradle plugin
- sbt-dependency-graph; see gitlab-org/security-products/analyzers/gemnasium-maven!107 (merged)
- all the dependency exports returned by a multi-project build are scanned
- only the root project is found by the
Finder
Implementation plan
-
change builders in gemnasium-maven
-
change the Builder
interface so that theBuild
function can return the dependency exports for the sub-projects, along with the dependency export for the root project gitlab-org/security-products/analyzers/gemnasium-maven!107 (merged) -
update sbt
builder: parse output ofsbt
and extract paths of DOT exports gitlab-org/security-products/analyzers/gemnasium-maven!107 (merged) -
change the Analyze
function so that additional exports returned by the builders are all processed gitlab-org/security-products/analyzers/gemnasium-maven!107 (merged) -
update maven
builder: parse output of Gemnasium Maven plugin and extract paths of JSON exports -
update gradle
builder: parse output of Gemnasium Gradle plugin and extract paths of JSON exports -
change finder
configuration and force the search mode toSearchSingleDir
; the preset will be updated later on
-
-
simplify; this will be done as part of #292952 (closed)finder
- in
gemnasium
, removeSearchSingleTree
mode and changegemnasium-maven
's preset toSearchSingleDir
- update
gemnasium
dependency ingemnasium-maven
, and revert changes to the preset
- in
NOTE: This introduces a discrepancy between the Builder
defined in gemnasium-maven with the one defined in gemnasium
and gemnasium-maven
, but this will be addressed when moving the code of gemnasium-maven
to gemnasium
. See #292952 (closed)
Improvements
- code is simplified, in particular the finder
- gemnasium-maven and gemnasium-python are further aligned and both use the
finder.SearchSingleDir
search mode - code is easier to debug; it no longer inspects directories that are not part of the multi-project build
Risks
- The location of vulnerabilities detected in sub-projects might change, resulting in the loss of vulnerability feedback.
- The
dependency_files[].path
for the sub-projects might change, in the report. This would only change the Dependency List page though. - Dependencies of the sub-projects might merged into the dependencies of the root project, the
dependency_files
field of the report. - In case their are multiple root projects in a repo, the new implementation might pick a different one. That said, there are no specs to establish what root project the analyzer should pick and scan.
Involved components
- https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium
- https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven
Optional: Intended side effects
Sub-projects can be scanned as long as they have a dependency file export, even if there's no build file is detected in the directory of the sub-project. This fixes a bug where Gradle sub-projects are skipped if the build file name isn't build.gradle
. See #322346 (closed)