Implement accurate dependency scanning for golang projects

feistelrequested to merge
feistel/gemnasium:feature/go-dep-scanning into master

What does this MR do?

Implement accurate dependency scanning for golang projects.

What's wrong with the current approach ?

  • dependency relations between modules are lost
  • false positives (doesn't handle multiple versions of the same module)
  • go.sum treated as a lock file

Goal

Dependency scanning should aim for a clear overview of your vulnerable dependencies. See gitlab-org/gitlab#227620 (closed)
This is not currently possible with a go project. The Dependency List tab shows each module as a direct dependency of the project and multiple versions are shown as additional dependencies (which is clearly wrong).

What's 'accurate' dependency scanning anyway ?

  • a list of packages used by the project (note: this is the final list of dependencies. Conflict resolution and any other resolution strategy have been evaluated at this point).
  • dependency relations between those packages are used to build the dependency graph

How ?

  1. Retrieve the list of modules used by the project with go list -m all and map each module to a specific version. Since this is the final list of dependencies so there are no duplicate modules (note: this is not uncommon, different modules can depend on different versions of the same module. Go uses an algorithm called Minimal version selection (MVS) to select a set of module versions to use when building packages, see https://golang.org/ref/mod#minimal-version-selection)
  2. Use go native command (go mod graph) to print the the module requirement graph (with replacements applied) in text form and parse the dependency relations (note: this is the dependency graph and multiple versions of the same module are allowed, attention should be paid to only use the version selected in the previous phase).

Caveats and gemnasium-go

Output of the above commands is generated and saved in a file named gemnasium-go (input wanted on the name).
Since this is not part of the go module system (unlike go.mod and go.sum), the file is generated by gemnasium using a builder.
Support for builders is added by !195 (closed), otherwise the file is not generated and the scanner is not able to find it.

What are the relevant issue numbers?

Closes gitlab-org/gitlab#229819 (closed)
Closes gitlab-org/gitlab#321081 (closed)
Depends on !195 (closed)

Does this MR meet the acceptance criteria?

Edited by feistel

Merge request reports