Gemnasium fails when running sbt dependencyDot: key not found: ModuleId(org.apache.commons,commons-io,1.3.2)
Summary
When scanning certain Sbt projects, the dependencyDot
task executed by the gemnasium-maven
analyzer fails with the following error:
java.util.NoSuchElementException: key not found: ModuleId(org.apache.commons,commons-io,1.3.2)
This has been reported for Sbt 1.6.1 projects, and it's been reproduced using Sbt 1.6.2.
Workaround
-
Add this line to
project/plugins.sbt
:addDependencyTreePlugin
-
Upgrade to sbt 1.7.2 or higher. This can be done by setting
sbt.version
inproject/build.properties
:sbt.version=1.7.2
See #390287 (comment 1350436313)
Further details
Based on the error stack, the error occurs when specific Sbt modules are evicted. The error might be triggered from https://github.com/sbt/sbt/blob/v1.6.1/main/src/main/scala/sbt/internal/graph/rendering/DOT.scala#L56.
org.apache.commons/commons-io/1.3.2
has moved to commons-io/commons-io/1.3.2
, which might trigger the bug.
dependencyDot
has been a built-in Sbt task starting from Sbt 1.4. It's used by gemnasium-maven
to export dependency graphs to DOT files.
Steps to reproduce
- Create an Sbt project that depends on
"com.freedomotic" % "freedomotic-core" % "5.6"
. - Add the dependencyTree plugin by
adding
addDependencyTreePlugin
toproject/plugins.sbt
. - Run
sbt dependencyDot
.
See #390287 (comment 1349807945)
Example Project
https://gitlab.com/gitlab-org/security-products/tests/scala-sbt/-/jobs/4100757706#L83
What is the current bug behavior?
gemnasium-maven-dependency_scanning
job fails when running sbt dependencyDot
.
What is the expected correct behavior?
gemnasium-maven-dependency_scanning
is successful.
Relevant logs and/or screenshots
[error] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[error] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[error] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[error] at java.base/java.lang.Thread.run(Thread.java:833)
[error] (Compile / dependencyDot / asString) java.util.NoSuchElementException: key not found: GraphModuleId(org.apache.commons,commons-io,1.3.2)
[error] Total time: 1 s, completed Apr 12, 2023, 12:01:26 PM
See #390287 (comment 1349807945)
Possible fixes
- Upgrade to sbt 1.7.2 or later, and enable the built-in sbt-dependency-tree by putting
addDependencyTreePlugin
inplugins.sbt
. This only works with sbt 1.4 and later. See #390287 (comment 1350432605). That doesn't seem to be compatible with offline environment though: gitlab-org/security-products/analyzers/gemnasium!487 (comment 1353966434) - Generate a graphml output. gitlab-org/security-products/analyzers/gemnasium!487 (comment 1357717810)
- Use sbt-bom and parse the CycloneDX XML SBOM it generates.
Proposal
See #390287 (comment 1438999474)
- for sbt 1.0.x: current implementation, no fix for this bug
- for sbt >= 1.1.x: new implementation based on sbt-sbom plugin that fixes the bug. See #390287 (comment 1428773956)
Previous proposal
Make gemnasium-maven write addDependencyTreePlugin
to user's plugins.sbt
when the sbt version is greater than 1.3.
We could also refactor gemnasium-maven and also make builder/sbt
responsible for configuring the plugin for sbt 1.3 and lower. Right now this is done when building the image. #390287 (comment 1351142721)
- Make gemnasium-maven write
addDependencyTreePlugin
to$HOME/.sbt/1.0/plugins/plugins.sbt
and/root/.sbt/1.0/plugins/plugins.sbt
when the version returned bysbt about
is greater that 1.3; look forThis is sbt X.Y.Z
. #390287 (comment 1351198948) - Add an image spec that triggers the problem described in this issue. #390287 (comment 1349719948)
That doesn't seem to be compatible with offline environment though: gitlab-org/security-products/analyzers/gemnasium!487 (comment 1353966434)
Implementation plan
- Add support for CycloneDX XML SBOM to
finder
. - Add file parser that extracts PURLs from CycloneDX XML file.
- Add the JAR of sbt-bom to the Docker image.
- Change sbt builder.
- Detect sbt version similar to gitlab-org/security-products/analyzers/gemnasium!487 (diffs).
- Call
sbt bom
instead ofsbt dependencyDot
if sbt 1.1 or higher.
- Update image specs.
- Add specs for relocated modules (sbt 1.1+). This is similar to gitlab-org/security-products/tests/scala-sbt!71 (closed).
- Adjust expectations for sbt 1.1+ if needed.
- NOTE: Specs for sbt 1.0.x don't change.
- Backport fix to Gemnasium
v3
- Update documentation of offline support.
- Document that it's necessary to run
sbt ivyReport
before going offline. - Make adjustments if needed.
- Document that it's necessary to run