Maven Gemnasium scanner does not use private maven repositories defined in build.gradle.kts in gemnasiumDumpDependencies task

Summary

In a multi-module maven project, the Maven Gemnasium dependency scanner fails to utilize the repositories defined for the subprojects in the backend:gemnasiumDumpDependencies task. The project structure has a frontend and a backend and both successfully build (and resolve dependencies) with gradlew assemble. This renders the scanner completely useless if referencing dependencies in private maven repositories.

Steps to reproduce

docker run -it --rm -e ADDITIONAL_CA_CERT_BUNDLE=<Certificate hidden for customer privacy> -e SECURE_LOG_LEVEL=debug -v /dev/gitlab/my-app:/my-app registry.gitlab.com/security-products/gemnasium-maven:3 /bin/bash

root@e1f08f82def6:/my-app# /analyzer run

Example Project

What is the current bug behavior?

The gemnasiumDumpDependencies task that is being injected into the build does not utilize the Maven private repositories defined for allprojects or subprojects.

What is the expected correct behavior?

The gemnasiumDumpDependencies task will utilize the privately hosted maven repository.

Relevant logs and/or screenshots

root@e1f08f82def6:/my-app# /analyzer run

Using java version 'adoptopenjdk-17.0.2+8'
[INFO] [gemnasium-maven] [2022-06-16T15:01:57Z] ▶ GitLab gemnasium-maven analyzer v3.2.0
[DEBU] [gemnasium-maven] [2022-06-16T15:01:58Z] ▶ /opt/asdf/shims/keytool -importcert -alias custom -file /etc/ssl/certs/ca-certificates.crt -trustcacerts -noprompt -storepass changeit -keystore /opt/asdf/installs/java/adoptopenjdk-17.0.2+8/lib/security/cacerts
Warning: use -cacerts option to access cacerts keystore
Certificate was added to keystore

[DEBU] [gemnasium-maven] [2022-06-16T15:01:58Z] ▶ inspect directory: .
[DEBU] [gemnasium-maven] [2022-06-16T15:01:58Z] ▶ skip ignored directory: .adcp
[DEBU] [gemnasium-maven] [2022-06-16T15:01:58Z] ▶ skip ignored directory: .git
[DEBU] [gemnasium-maven] [2022-06-16T15:01:58Z] ▶ skip ignored directory: .github
[DEBU] [gemnasium-maven] [2022-06-16T15:01:58Z] ▶ skip ignored directory: .gradle
[DEBU] [gemnasium-maven] [2022-06-16T15:01:58Z] ▶ electing gradle for maven because this is the first match
[DEBU] [gemnasium-maven] [2022-06-16T15:01:58Z] ▶ electing gradle for maven because this is the first match
[INFO] [gemnasium-maven] [2022-06-16T15:01:58Z] ▶ Detected supported dependency files in '.'. Dependency files detected in this directory will be processed. Dependency files in other directories will be skipped.
[DEBU] [gemnasium-maven] [2022-06-16T15:01:58Z] ▶ Exporting dependencies for /my-app/build.gradle.kts
[DEBU] [gemnasium-maven] [2022-06-16T15:03:21Z] ▶ /my-app/gradlew --init-script /gemnasium-gradle-plugin-init.gradle gemnasiumDumpDependencies
Downloading https://services.gradle.org/distributions/gradle-7.3.1-bin.zip
...........10%...........20%...........30%...........40%...........50%...........60%...........70%...........80%...........90%...........100%

Welcome to Gradle 7.3.1!

Here are the highlights of this release:
 - Easily declare new test suites in Java projects
 - Support for Java 17
 - Support for Scala 3

For more details see https://docs.gradle.org/7.3.1/release-notes.html

Starting a Gradle Daemon (subsequent builds will be faster)

> Configure project :backend
{PATH=/opt/asdf/shims:/opt/asdf/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin, GEMNASIUM_DB_REF_NAME=master, ASDF_DIR=/opt/asdf, GRADLE_PLUGIN_INIT_PATH=/gemnasium-gradle-plugin-init.gradle, CI_DEBUG_TRACE=false, ASDF_JAVA_VERSION=adoptopenjdk-17.0.2+8, HISTSIZE=0, JAVA_HOME=/opt/asdf/installs/java/adoptopenjdk-17.0.2+8, DEP_GRAPH_PLUGIN=addSbtPlugin("net.virtual-void" % "sbt-dependency-graph" % "0.10.0-RC1" from "file:///sbt-dependency-graph-plugin/sbt-dependency-graph-0.10.0-RC1.jar"), TERM=xterm, LANG=C.UTF-8, HISTFILESIZE=0, SECURE_LOG_LEVEL=debug, PROMPT_COMMAND=prompt_command; prompt_command, PWD=/my-app, _=/analyzer-binary, ASDF_VERSION=v0.8.1, GEMNASIUM_DB_REMOTE_URL=https://gitlab.com/gitlab-org/security-products/gemnasium-db.git, OLDPWD=/my-app, ASDF_DATA_DIR=/opt/asdf, HOSTNAME=e1f08f82def6, GEMNASIUM_DB_LOCAL_PATH=/gemnasium-db, ADDITIONAL_CA_CERT_BUNDLE=<Certificate hidden for customer privacy>, SHLVL=2, HOME=/gemnasium-maven, GEMNASIUM_DB_WEB_URL=https://gitlab.com/gitlab-org/security-products/gemnasium-db, VRANGE_DIR=/vrange}

> Task :gemnasiumDumpDependencies
No dependencies found in project

> Task :acceptance:gemnasiumDumpDependencies
No dependencies found in project

> Task :backend:gemnasiumDumpDependencies FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':backend:gemnasiumDumpDependencies'.
> Project has 1 unresolved dependencies: **(Dependency name hidden but confirmed to be accessible in the private maven repository)**

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

See https://docs.gradle.org/7.3.1/userguide/command_line_interface.html#sec:command_line_warnings

BUILD FAILED in 1m 22s
3 actionable tasks: 3 executed

exit status 1

Possible fixes

See #365484 (comment 1066679855)

Contributing factor(s)

The keytool does not import all certificates in a certificate bundle. This can be tested by running the following steps using the latest gemnasium-maven v3 image i.e. registry.gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven:3.

Note: These steps emulate the steps taken by the analyzer

The analyzer appends the contents of ADDITIONAL_CA_CERT_BUNDLE to the default ca bundle path i.e. /etc/ssl/certs/ca-certificates.crt: code ref. This is the equivalent of the following: cat $ADDITIONAL_CA_CERT_BUNDLE >> /etc/ssl/certs/ca-certificates.crt

The analyzer then updates the default java keystore by importing the default ca bundle path: code ref. This is the equivalent of the following:

keytool -importcert \
-alias custom \
-file /etc/ssl/certs/ca-certficates.crt
-trustcacerts \
-noprompt \
-storepass changeit \
-keystore "$JAVA_HOME/lib/security/cacerts"

The analyzer then runs the dependency scan with only the first certificate inside /etc/ssl/certs/ca-certificates.crt added. This can be verified by outputting the contents of keystore and comparing against the first certificate in the default bundle path.
```
keytool -list -noprompt -alias custom -keystore $JAVA_HOME/lib/security/cacerts -storepass changeit
head -n 100 /etc/ssl/certs/ca-certificates.crt
```

Note: You can also verify that only one certficate was added by doing a before and after comparison of the total certs inside of the keystore. This requires ommitting the -alias custom flag argument in the 3^rd step.

Implementation plan

The bug fix requires adding every certificate individually instead of all at once.

update the Update function to so that it splits the bundle into individual certs and then adds them sequentially.

Out of scope

Add integration tests to test the analyzer with private repos. See Add integration tests that use a private maven ... (#380871 - closed)

Edited Nov 09, 2022 by Oscar Tovar