Replace gemnasium-gradle-plugin with the htmlDependencyReport task in gemnasium-maven
Problem to solve
gemnasium-maven currently uses the gemnasium-gradle-plugin to generate the dependency list for a project. The gemnasium-gradle-plugin
presents the following challenges:
- It's written in
kotlin
, and there are nokotlin
experts on the Composition Analysis team - If changes are made to the plugin, it requires a new release of the plugin, as well as updating
gemnasium-maven
to use the new plugin - It's not easy to test the plugin with
gemnasium-maven
, since we need to publish a new version before we can make use of it - It requires us to maintain and install a separate non-standard project
- There are limitations that prevent it from working on all projects
Instead of using the gemnasium-gradle-plugin
, we can leverage the htmlDependencyReport
task provided by the Project Report Plugin to generate the full dependency tree. The htmlDependencyReport
is part of the standard Gradle distribution, which gives us the following benefits:
- We don't need to maintain the code ourselves
- It ships with Gradle, so we don't need to package and install a separate non-standard project
- It produces JSON output
- It supports more projects than the
gemnasium-gradle-plugin
Because of these advantages, we should replace the gemnasium-gradle-plugin
with the htmlDependencyReport
task, which is the purpose of this issue.
Follow-up: run the androidDependencies
task to implement Android support. See #336866 (closed)
Proposal
Update gemnasium-maven
so that it uses the htmlDependencyReport
task to generate the dependency tree. This will require us to update the mvnplugin.Parse function to handle the JSON
output by the htmlDependencyReport
task.
Further details
JSON files are generated by the JsonProjectDependencyRenderer class.
Intended users
User experience goal
Projects are analyzed for dependencies the same way they currently are, but we support more projects.
Current implementation
Right now gemnasium-maven
has a gradle
builder that runs the gemnasiumDependenciesTask
handled by the Gemnasium Maven Plugin. The task creates a JSON file that lists the dependencies of all Gradle dependency configurations, and this file is parsed by the mvnplugin file parser. The parser simply collects the package names and versions; it doesn't return a dependency graph.
htmlDependencyReport
task
The htmlDependencyReport
task provided by the Project Report Plugin is shipped with the Gradle source. In order to make use of it, we need to add the project-report
plugin by creating a Gradle Initialization Script with the following contents:
allprojects { apply plugin: 'project-report' }
Assuming we've placed the above commands into a file named init.gradle
, we can then run the htmlDependencyreport
task as follows:
gradle --init-script init.gradle htmlDependencyReport
This command will output a js
file for each sub-project detected within the given project, and produces the following output:
> Task :htmlDependencyReport
See the report at: file:///src/build/reports/project/dependencies/index.html
> Task :mail:protocols:pop3:htmlDependencyReport
See the report at: file:///src/mail/protocols/pop3/build/reports/project/dependencies/index.html
> Task :plugins:openpgp-api-lib:openpgp-api:htmlDependencyReport
See the report at: file:///src/plugins/openpgp-api-lib/openpgp-api/build/reports/project/dependencies/index.html
The js
files for the above sub-projects are located in the following files:
build/reports/project/dependencies/root.js
mail/protocols/pop3/build/reports/project/dependencies/root.mail.protocols.pop3.js
plugins/openpgp-api-lib/openpgp-api/build/reports/project/dependencies/root.plugins.openpgp-api-lib.openpgp-api.js
We can convert the Task :<sub-project-name>:htmlDependencyReport
string from the above text output into a js
file path as follows:
func taskStringToPath(taskStr string) string {
taskStr = strings.TrimPrefix(taskStr, "Task :")
taskStr = strings.TrimRight(taskStr, ":htmlDependencyReport")
fileName := strings.Replace(taskStr, ":", ".", -1)
filePrefix := strings.Replace(taskStr, ":", "/", -1)
return fmt.Sprintf("%s/build/reports/project/dependencies/root.%s.js", filePrefix, fileName)
}
This js
file has the following structure and contains the dependency tree in JSON
format:
var projectDependencyReport = {"gradleVersion":"Gradle 7.4","generationDate":"Wed Mar 30 06:48:21 UTC 2022","project":{"name":"plugins","description":null,"configurations":[{"name":"ktlint","description":"Main ktlint-gradle configuration","dependencies":[{"module":"com.pinterest:ktlint","name":"com.pinterest:ktlint:0.40.0","resolvable":"RESOLVED","hasConflict":false}]}]}};
If we remove the JavaScript preamble var projectDependencyReport =
from the above js
file, as well as the trailing semicolon ;
, then we end up with valid JSON
.
See here for an example of running the htmlDependencyReport
task and the resulting JSON is available here.
By default, the htmlDependencyReport
task lists all dependencies of all configurations (scopes), which means that we'll need to de-duplicate these entries. See Listing dependencies in a project:
Every Gradle project provides the task dependencies to render the so-called dependency report from the command line. By default the dependency report renders dependencies for all configurations.
Multi-project build
In a multi-project Gradle build, the htmlDependencyReport
task lists the dependencies of all sub-projects.
Documentation
Add documentation to explain that gemnasium-maven
uses the htmlDependencyReport
task provided by the Project Report Plugin to generate the dependency tree.
Testing
gemnasium-maven
has multiple image tests for Gradle. The new feature added by this issue must not break any of these existing tests.
What does success look like, and how can we measure that?
A dependency list can be generated using the htmlDependencyReport
task without breaking existing tests.
We also believe this will resolve our android issues - and should validate asap (while in testing, pre-release) with @Regis.Guillermin and any other customers we become aware of who are using android and willing to test an image. (do not wait on customer feedback if this works for the test cases however). See #336866 (closed)
What is the type of buyer?
Enterprise Edition GitLab Ultimate
Is this a cross-stage feature?
No, this only affects groupcomposition analysis
Implementation plan
-
Add a new init.gradle
file togemnasium-maven
, and update the Dockerfile to copy this file to the/
directory in the Docker image. This file must contain the following:allprojects { apply plugin: 'project-report' }
This will enable the Project Report Plugin and allow us to export a
JSON
dependency list. -
Update the --init-script
argument to refer to theinit.gradle
initialization script added in step1.
above, and use thehtmlDependencyReport
task instead of thegemnasiumDumpDependencies
task:initScript := "init.gradle" taskName := "htmlDependencyReport" args = append(args, "--init-script", initScript, taskName)
-
Create a new extractor in the gemnasium-maven exportpath
package to parse the output of thehtmlDependencyReport
task executed above, to extract the paths to theJSON
files for each sub-project. For example, assuming the output is the following:root@d04f6cdbcf81:/src# gradle --init-script init.gradle htmlDependencyReport > Task :htmlDependencyReport See the report at: file:///src/build/reports/project/dependencies/index.html > Task :mail:protocols:pop3:htmlDependencyReport See the report at: file:///src/mail/protocols/pop3/build/reports/project/dependencies/index.html > Task :plugins:openpgp-api-lib:openpgp-api:htmlDependencyReport See the report at: file:///src/plugins/openpgp-api-lib/openpgp-api/build/reports/project/dependencies/index.html
We should obtain the following list of
JSON
files for the sub-projects:build/reports/project/dependencies/root.js
mail/protocols/pop3/build/reports/project/dependencies/root.mail.protocols.pop3.js
plugins/openpgp-api-lib/openpgp-api/build/reports/project/dependencies/root.plugins.openpgp-api-lib.openpgp-api.js
-
Implement a new file parser in the gemnasium
project to parse the dependency listJSON
files produced in step3.
above.Here's an example of the dependency JSON structure: dependencies.json
We'll need to make sure we handle the following edge cases:
-
We need to properly parse dependencies that had to undergo conflict resolution, and therefore the requested and selected versions are separated by a right arrow character
➡
(unicode characterU+27A1
). An example of this situation can be found in this fixture file and the version oforg.slf4j/slf4j-api
that it resolves to in the gl-dependency-scanning-report.json expectation. -
Handle the situation where
gradle dependencies
fails to resolve a dependency, for example:$ git clone git@gitlab.com:gitlab-org/security-products/analyzers/gemnasium-gradle-plugin.git && cd gemnasium-gradle-plugin && git checkout 42c5ff41c25fceb561480b379f1886034a09d303 $ docker run -it --rm -e SECURE_LOG_LEVEL=debug -v "$PWD:/gemnasium-gradle-plugin-src" -w /gemnasium-gradle-plugin/src registry.gitlab.com/security-products/gemnasium-maven:2.27.4 bash -ic 'gradle -p /gemnasium-gradle-plugin-src dependencies' Welcome to Gradle 6.7.1! ... > Task :dependencies ... functionalTestImplementationDependenciesMetadata +--- org.jetbrains.kotlin:kotlin-stdlib:1.5.10 ... +--- org.jetbrains.kotlin:kotlin-test:1.5.10 FAILED +--- org.jetbrains.kotlin:kotlin-test-junit:1.5.10 | +--- org.jetbrains.kotlin:kotlin-test:1.5.10 FAILED
Notice the
org.jetbrains.kotlin:kotlin-test:1.5.10
dependency failed to be resolved. We need to make sure to address this situation.
This step is to be implemented in Implement gemnasium parser for gradle dependenc... (#360626 - closed).
-
-
Add unit tests for the new file parser added in step 4.
above, similar to the currentgradle-dependencies.json
test. -
Improve image integration test coverage: -
Add a test with a single dependency with different versions in different scopes (configurations). For example: dependencies { testRuntimeClasspath 'org.slf4j:slf4j-api:1.7.31' implementation 'org.slf4j:slf4j-api:1.7.30' runtimeClasspath 'org.slf4j:slf4j-api:1.7.25' }
-
Add a test to handle gradle dependency constraints as described in this comment. To be more specific, we must be able to show that we can successfully execute a dependency scan against the files in this branch. -
Add a test to handle the the situation where gemnasium-maven
returns the wrong version of a dependency that underwent conflict resolution, as explained in this comment. -
Add a test to handle the the situation where gradle dependencies
fails to resolve a dependency, as described in step4.
above. -
Add a test with large dependency files, such as the 316 MB
file discussed here -
Add a test against a large open-source gradle project, such as k9mail/k-9. See this comment for more details
-
-
Update the Dependency Scanning Documentation to include details explaining that gemnasium-maven
uses thehtmlDependencyReport
task from the Project Report Plugin to generate dependency information. -
Remove gemnasium-gradle-plugin parsing code from gemnasium
:-
Remove gradle-dependencies.json
frommvnplugin.go
-
Remove gemnasium-gradle-plugin
references from thegemnasium-maven
Dockerfile -
Remove gradle plugin installer from the config/install.sh script
-