Skip to content

Gemnasium fails when running sbt dependencyDot: key not found: ModuleId(org.apache.commons,commons-io,1.3.2)

Summary

When scanning certain Sbt projects, the dependencyDot task executed by the gemnasium-maven analyzer fails with the following error:

java.util.NoSuchElementException: key not found: ModuleId(org.apache.commons,commons-io,1.3.2)

This has been reported for Sbt 1.6.1 projects, and it's been reproduced using Sbt 1.6.2.

Workaround

  1. Add this line to project/plugins.sbt:

    addDependencyTreePlugin

  2. Upgrade to sbt 1.7.2 or higher. This can be done by setting sbt.version in project/build.properties:

    sbt.version=1.7.2

See #390287 (comment 1350436313)

Further details

Based on the error stack, the error occurs when specific Sbt modules are evicted. The error might be triggered from https://github.com/sbt/sbt/blob/v1.6.1/main/src/main/scala/sbt/internal/graph/rendering/DOT.scala#L56.

org.apache.commons/commons-io/1.3.2 has moved to commons-io/commons-io/1.3.2, which might trigger the bug.

dependencyDot has been a built-in Sbt task starting from Sbt 1.4. It's used by gemnasium-maven to export dependency graphs to DOT files.

Steps to reproduce

  • Create an Sbt project that depends on "com.freedomotic" % "freedomotic-core" % "5.6".
  • Add the dependencyTree plugin by adding addDependencyTreePlugin to project/plugins.sbt.
  • Run sbt dependencyDot.

See #390287 (comment 1349807945)

Example Project

https://gitlab.com/gitlab-org/security-products/tests/scala-sbt/-/jobs/4100757706#L83

What is the current bug behavior?

gemnasium-maven-dependency_scanning job fails when running sbt dependencyDot.

What is the expected correct behavior?

gemnasium-maven-dependency_scanning is successful.

Relevant logs and/or screenshots

[error] 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[error] 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[error] 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[error] 	at java.base/java.lang.Thread.run(Thread.java:833)
[error] (Compile / dependencyDot / asString) java.util.NoSuchElementException: key not found: GraphModuleId(org.apache.commons,commons-io,1.3.2)
[error] Total time: 1 s, completed Apr 12, 2023, 12:01:26 PM

See #390287 (comment 1349807945)

Possible fixes

Proposal

See #390287 (comment 1438999474)

  • for sbt 1.0.x: current implementation, no fix for this bug
  • for sbt >= 1.1.x: new implementation based on sbt-sbom plugin that fixes the bug. See #390287 (comment 1428773956)
Previous proposal

Make gemnasium-maven write addDependencyTreePlugin to user's plugins.sbt when the sbt version is greater than 1.3.

We could also refactor gemnasium-maven and also make builder/sbt responsible for configuring the plugin for sbt 1.3 and lower. Right now this is done when building the image. #390287 (comment 1351142721)

  • Make gemnasium-maven write addDependencyTreePlugin to $HOME/.sbt/1.0/plugins/plugins.sbt and /root/.sbt/1.0/plugins/plugins.sbt when the version returned by sbt about is greater that 1.3; look for This is sbt X.Y.Z. #390287 (comment 1351198948)
  • Add an image spec that triggers the problem described in this issue. #390287 (comment 1349719948)

That doesn't seem to be compatible with offline environment though: gitlab-org/security-products/analyzers/gemnasium!487 (comment 1353966434)

Implementation plan

Edited by Fabien Catteau