Dependency Scanning failed for sbt project with error "panic: simple: adding self edge [recovered]"
Summary
Using dependency scanning template for a sbt project failed with error "panic: simple: adding self edge [recovered]"
include:
- template: Dependency-Scanning.gitlab-ci.yml
variables:
SECURE_LOG_LEVEL: debug
A similar error might happen when processing some yarn projects. In that case the error originates form convert.NewGraph
, when generating the Dependency Scanning report.
Further details
The gemnasium-maven analyzer runs the dependencyDot
task of the sbt-dependency-graph plugin to export dependency graphs of Sbt projects to DOT files, and then parses DOT files using gonum. More specifically, the DOT file parsers creates a gonum/graph/simple.DirectedGraph
, and then decode it using gonum/graph/encoding/dot
. See https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium/-/blob/v2.29.9/scanner/parser/sbt/dotgraph.go#L18
Unfortunately, DirectedGraph.SetEdge doesn't support self edges:
SetEdge adds e, an edge from one node to another. If the nodes do not exist, they are added and are set to the nodes of the edge otherwise. It will panic if the IDs of the e.From and e.To are equal.
Proposal
Define dotGraph.SetEdge
so that it does nothing when the argument is a self edge. If not, then it simply delegates to the SetEdge
method of the embedded directed graph.
func (g *dotGraph) SetEdge(e graph.Edge) {
if e.From() == e.To() {
return
}
g.DirectedGraph.SetEdge(e)
}
// dotGraph is the simple.DirectedGraph representation of a dot file
type dotGraph struct {
*simple.DirectedGraph
}
Similarly, convert.NewGraph
must be updated in order to skip self-edges when building the dependency graph.
An alternative would be to use a another type of graph that supports self edges, but we could have problems when processing these self edges further down the road in the Gemnasium analyzer.
In any case, we should investigate and figure out what causes these self edges, and what are the implications for features that rely on the dependency graph. See #333254 (comment 618859518)
Implementation plan
-
Update file parser in gemnasium
project.- Change
sbt
file parser, to skip self-edges when parsing the DOT graph. - Change
convert.NewGraph
, to skip self-edges when building the graph. - Release a new PATCH version.
- Change
-
Update gemnasium-maven
.- Update
gemnasium
git submodule. - Release a new PATCH version.
- Update
gemnasium-python doesn't seem to be impacted by this, but it might be safer to release a new version anyway, after upgrading to the latest version of gemnasium (git submodule).
Steps to reproduce
- sbt-dependency-graph and sbt-dependency-check settings are configured in plugins.sbt and build.sbt (for local usage)
- Include dependency scanning template
- Run the pipeline
Example Project
What is the current bug behavior?
Job gemnasium-maven-dependency_scanning
failed with error "panic: simple: adding self edge [recovered]". No gl-dependency-scanning-report.json
file is generated
What is the expected correct behavior?
Job gemnasium-maven-dependency_scanning
is expected to succeed with gl-dependency-scanning-report.json
file generated
Relevant logs and/or screenshots
Job console log
[info] welcome to sbt 1.4.3 (AdoptOpenJDK Java 11.0.7)
[info] loading settings for project global-plugins from plugins.sbt ...
[info] loading global plugins from /root/.sbt/1.0/plugins
[info] loading settings for project map-update-pipeline-build from assembly.sbt,plugins.sbt ...
[info] loading project definition from /builds/ce/content-management/content-sourcing/map-update-pipeline/project
[info] loading settings for project root from build.sbt ...
[info] set current project to Map Updating Pipeline (in build file:/builds/ce/content-management/content-sourcing/map-update-pipeline/)
[warn] there's a key that's not used by any other settings/tasks:
[warn]
[warn] * root / dependencyCheck / logLevel
[warn] +- /builds/ce/content-management/content-sourcing/map-update-pipeline/build.sbt:10
[warn]
[warn] note: a setting might still be used by a command; to exclude a key from this `lintUnused` check
[warn] either append it to `Global / excludeLintKeys` or call .withRank(KeyRanks.Invisible) on the key
[warn] There may be incompatibilities among your library dependencies; run 'evicted' to see detailed eviction warnings.
[info] Wrote dependency graph to '/builds/ce/content-management/content-sourcing/map-update-pipeline/target/dependencies-compile.dot'
[success] Total time: 3 s, completed Jun 10, 2021, 11:05:33 AM
[DEBU] [gemnasium-maven] [2021-06-10T11:05:34Z] ▶ Exporting dependencies for /builds/ce/content-management/content-sourcing/map-update-pipeline/lambdaLayer/build.sbt
[DEBU] [gemnasium-maven] [2021-06-10T11:05:54Z] ▶ /opt/asdf/shims/sbt dependencyDot
[info] Loading settings for project global-plugins from plugins.sbt ...
[info] Loading global plugins from /root/.sbt/1.0/plugins
[info] Loading settings for project lambdalayer-build from plugins.sbt ...
[info] Loading project definition from /builds/ce/content-management/content-sourcing/map-update-pipeline/lambdaLayer/project
[warn] There may be incompatibilities among your library dependencies; run 'evicted' to see detailed eviction warnings.
[info] Loading settings for project root from build.sbt ...
[info] Set current project to root (in build file:/builds/ce/content-management/content-sourcing/map-update-pipeline/lambdaLayer/)
[warn] There may be incompatibilities among your library dependencies; run 'evicted' to see detailed eviction warnings.
[info] Wrote dependency graph to '/builds/ce/content-management/content-sourcing/map-update-pipeline/lambdaLayer/target/dependencies-compile.dot'
[success] Total time: 1 s, completed Jun 10, 2021, 11:05:54 AM
panic: simple: adding self edge [recovered]
panic: simple: adding self edge
goroutine 1 [running]:
gonum.org/v1/gonum/graph/encoding/dot.copyGraph.func1(0xc000220a70)
/go/pkg/mod/gonum.org/v1/gonum@v0.8.1/graph/encoding/dot/decode.go:87 +0xb0
panic(0x83b2a0, 0x922c30)
/usr/local/go/src/runtime/panic.go:969 +0x1b9
gonum.org/v1/gonum/graph/simple.(*DirectedGraph).SetEdge(0xc0004ca270, 0x931f60, 0xc0001f11e0)
/go/pkg/mod/gonum.org/v1/gonum@v0.8.1/graph/simple/directed.go:189 +0x4ed
gonum.org/v1/gonum/graph/encoding/dot.(*simpleGraph).addEdgeStmt(0xc0002209c8, 0x935520, 0xc000550028, 0xc000193080)
/go/pkg/mod/gonum.org/v1/gonum@v0.8.1/graph/encoding/dot/decode.go:281 +0x1fd
gonum.org/v1/gonum/graph/encoding/dot.(*simpleGraph).addStmt(0xc0002209c8, 0x935520, 0xc000550028, 0x92fd20, 0xc000193080)
/go/pkg/mod/gonum.org/v1/gonum@v0.8.1/graph/encoding/dot/decode.go:197 +0x185
gonum.org/v1/gonum/graph/encoding/dot.copyGraph(0x935520, 0xc000550028, 0xc000193a70, 0x0, 0x0)
/go/pkg/mod/gonum.org/v1/gonum@v0.8.1/graph/encoding/dot/decode.go:103 +0x199
gonum.org/v1/gonum/graph/encoding/dot.Unmarshal(0xc00013e000, 0xc75d, 0xfe00, 0x935520, 0xc000550028, 0x5, 0xc00013e000)
/go/pkg/mod/gonum.org/v1/gonum@v0.8.1/graph/encoding/dot/decode.go:53 +0x95
gitlab.com/gitlab-org/security-products/analyzers/gemnasium/v2/scanner/parser/sbt.newDotGraph(0xc00013e000, 0xc75d, 0xfe00, 0xc00013e000, 0xc75d, 0xfe00)
/go/pkg/mod/gitlab.com/gitlab-org/security-products/analyzers/gemnasium/v2@v2.29.6/scanner/parser/sbt/dotgraph.go:18 +0x132
gitlab.com/gitlab-org/security-products/analyzers/gemnasium/v2/scanner/parser/sbt.Parse(0x92e540, 0xc000550020, 0xc00017e2d8, 0xc00017e388, 0x852900, 0xc000101890, 0xc00017e2c0, 0xc00017e2c0, 0x0, 0x0)
/go/pkg/mod/gitlab.com/gitlab-org/security-products/analyzers/gemnasium/v2@v2.29.6/scanner/parser/sbt/sbt.go:63 +0x8e
gitlab.com/gitlab-org/security-products/analyzers/gemnasium/v2/scanner.parseReader(0x92e540, 0xc000550020, 0x8ca468, 0x8af253, 0x3, 0x8b005d, 0x5, 0xc00006b3b0, 0x1, 0x1, ...)
/go/pkg/mod/gitlab.com/gitlab-org/security-products/analyzers/gemnasium/v2@v2.29.6/scanner/scanner.go:163 +0x55
gitlab.com/gitlab-org/security-products/analyzers/gemnasium/v2/scanner.Scanner.scanReader(0xc00001c378, 0xd, 0xc00001a258, 0x40, 0xc00001a0d5, 0x3c, 0xc0000182b6, 0x6, 0x92e540, 0xc000550020, ...)
/go/pkg/mod/gitlab.com/gitlab-org/security-products/analyzers/gemnasium/v2@v2.29.6/scanner/scanner.go:150 +0xf9
gitlab.com/gitlab-org/security-products/analyzers/gemnasium/v2/scanner.Scanner.ScanFile(0xc00001c378, 0xd, 0xc00001a258, 0x40, 0xc00001a0d5, 0x3c, 0xc0000182b6, 0x6, 0xc0001fe000, 0x67, ...)
/go/pkg/mod/gitlab.com/gitlab-org/security-products/analyzers/gemnasium/v2@v2.29.6/scanner/scanner.go:142 +0x18d
gitlab.com/gitlab-org/security-products/analyzers/gemnasium/v2/scanner.Scanner.ScanProjects(0xc00001c378, 0xd, 0xc00001a258, 0x40, 0xc00001a0d5, 0x3c, 0xc0000182b6, 0x6, 0xc00001e8c0, 0x42, ...)
/go/pkg/mod/gitlab.com/gitlab-org/security-products/analyzers/gemnasium/v2@v2.29.6/scanner/scanner.go:118 +0x3ea
main.analyze(0xc00003af80, 0xc00001e8c0, 0x42, 0x1, 0x1, 0x0, 0x1)
/go/src/app/analyze.go:102 +0x3f7
gitlab.com/gitlab-org/security-products/analyzers/command.Run.func1(0xc00003af80, 0x0, 0x0)
/go/pkg/mod/gitlab.com/gitlab-org/security-products/analyzers/command@v1.1.0/run.go:137 +0x559
github.com/urfave/cli/v2.(*Command).Run(0xc000136480, 0xc00003aa00, 0x0, 0x0)
/go/pkg/mod/github.com/urfave/cli/v2@v2.3.0/command.go:163 +0x4ed
github.com/urfave/cli/v2.(*App).RunContext(0xc000105380, 0x932be0, 0xc000020110, 0xc00000c080, 0x2, 0x2, 0x0, 0x0)
/go/pkg/mod/github.com/urfave/cli/v2@v2.3.0/app.go:313 +0x81f
github.com/urfave/cli/v2.(*App).Run(...)
/go/pkg/mod/github.com/urfave/cli/v2@v2.3.0/app.go:224
main.main()
/go/src/app/main.go:36 +0x379
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
(For installations with omnibus-gitlab package run and paste the output of: `sudo gitlab-rake gitlab:env:info`) (For installations from source run and paste the output of: `sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)