Replace gemnasium-gradle-plugin with the htmlDependencyReport task in gemnasium-maven

Problem to solve

gemnasium-maven currently uses the gemnasium-gradle-plugin to generate the dependency list for a project. The gemnasium-gradle-plugin presents the following challenges:

  1. It's written in kotlin, and there are no kotlin experts on the Composition Analysis team
  2. If changes are made to the plugin, it requires a new release of the plugin, as well as updating gemnasium-maven to use the new plugin
  3. It's not easy to test the plugin with gemnasium-maven, since we need to publish a new version before we can make use of it
  4. It requires us to maintain and install a separate non-standard project
  5. There are limitations that prevent it from working on all projects

Instead of using the gemnasium-gradle-plugin, we can leverage the htmlDependencyReport task provided by the Project Report Plugin to generate the full dependency tree. The htmlDependencyReport is part of the standard Gradle distribution, which gives us the following benefits:

  1. We don't need to maintain the code ourselves
  2. It ships with Gradle, so we don't need to package and install a separate non-standard project
  3. It produces JSON output
  4. It supports more projects than the gemnasium-gradle-plugin

Because of these advantages, we should replace the gemnasium-gradle-plugin with the htmlDependencyReport task, which is the purpose of this issue.

Follow-up: run the androidDependencies task to implement Android support. See #336866 (closed)

Proposal

Update gemnasium-maven so that it uses the htmlDependencyReport task to generate the dependency tree. This will require us to update the mvnplugin.Parse function to handle the JSON output by the htmlDependencyReport task.

Further details

JSON files are generated by the JsonProjectDependencyRenderer class.

Intended users

User experience goal

Projects are analyzed for dependencies the same way they currently are, but we support more projects.

Current implementation

Right now gemnasium-maven has a gradle builder that runs the gemnasiumDependenciesTask handled by the Gemnasium Maven Plugin. The task creates a JSON file that lists the dependencies of all Gradle dependency configurations, and this file is parsed by the mvnplugin file parser. The parser simply collects the package names and versions; it doesn't return a dependency graph.

htmlDependencyReport task

The htmlDependencyReport task provided by the Project Report Plugin is shipped with the Gradle source. In order to make use of it, we need to add the project-report plugin by creating a Gradle Initialization Script with the following contents:

allprojects { apply plugin: 'project-report' }

Assuming we've placed the above commands into a file named init.gradle, we can then run the htmlDependencyreport task as follows:

gradle --init-script init.gradle htmlDependencyReport

This command will output a js file for each sub-project detected within the given project, and produces the following output:

> Task :htmlDependencyReport
See the report at: file:///src/build/reports/project/dependencies/index.html

> Task :mail:protocols:pop3:htmlDependencyReport
See the report at: file:///src/mail/protocols/pop3/build/reports/project/dependencies/index.html

> Task :plugins:openpgp-api-lib:openpgp-api:htmlDependencyReport
See the report at: file:///src/plugins/openpgp-api-lib/openpgp-api/build/reports/project/dependencies/index.html

The js files for the above sub-projects are located in the following files:

  • build/reports/project/dependencies/root.js
  • mail/protocols/pop3/build/reports/project/dependencies/root.mail.protocols.pop3.js
  • plugins/openpgp-api-lib/openpgp-api/build/reports/project/dependencies/root.plugins.openpgp-api-lib.openpgp-api.js

We can convert the Task :<sub-project-name>:htmlDependencyReport string from the above text output into a js file path as follows:

func taskStringToPath(taskStr string) string {
  taskStr = strings.TrimPrefix(taskStr, "Task :")
  taskStr = strings.TrimRight(taskStr, ":htmlDependencyReport")

  fileName := strings.Replace(taskStr, ":", ".", -1)
  filePrefix := strings.Replace(taskStr, ":", "/", -1)

  return fmt.Sprintf("%s/build/reports/project/dependencies/root.%s.js", filePrefix, fileName)
}

This js file has the following structure and contains the dependency tree in JSON format:

var projectDependencyReport = {"gradleVersion":"Gradle 7.4","generationDate":"Wed Mar 30 06:48:21 UTC 2022","project":{"name":"plugins","description":null,"configurations":[{"name":"ktlint","description":"Main ktlint-gradle configuration","dependencies":[{"module":"com.pinterest:ktlint","name":"com.pinterest:ktlint:0.40.0","resolvable":"RESOLVED","hasConflict":false}]}]}};

If we remove the JavaScript preamble var projectDependencyReport = from the above js file, as well as the trailing semicolon ;, then we end up with valid JSON.

See here for an example of running the htmlDependencyReport task and the resulting JSON is available here.

By default, the htmlDependencyReport task lists all dependencies of all configurations (scopes), which means that we'll need to de-duplicate these entries. See Listing dependencies in a project:

Every Gradle project provides the task dependencies to render the so-called dependency report from the command line. By default the dependency report renders dependencies for all configurations.

Multi-project build

In a multi-project Gradle build, the htmlDependencyReport task lists the dependencies of all sub-projects.

Documentation

Add documentation to explain that gemnasium-maven uses the htmlDependencyReport task provided by the Project Report Plugin to generate the dependency tree.

Testing

gemnasium-maven has multiple image tests for Gradle. The new feature added by this issue must not break any of these existing tests.

What does success look like, and how can we measure that?

A dependency list can be generated using the htmlDependencyReport task without breaking existing tests.

We also believe this will resolve our android issues - and should validate asap (while in testing, pre-release) with @Regis.Guillermin and any other customers we become aware of who are using android and willing to test an image. (do not wait on customer feedback if this works for the test cases however). See #336866 (closed)

What is the type of buyer?

Enterprise Edition GitLab Ultimate

Is this a cross-stage feature?

No, this only affects groupcomposition analysis

Implementation plan

  1. Add a new init.gradle file to gemnasium-maven, and update the Dockerfile to copy this file to the / directory in the Docker image. This file must contain the following:

    allprojects { apply plugin: 'project-report' }

    This will enable the Project Report Plugin and allow us to export a JSON dependency list.

  2. Update the --init-script argument to refer to the init.gradle initialization script added in step 1. above, and use the htmlDependencyReport task instead of the gemnasiumDumpDependencies task:

    initScript := "init.gradle"
taskName := "htmlDependencyReport"
args = append(args, "--init-script", initScript, taskName)

  3. Create a new extractor in the gemnasium-maven exportpath package to parse the output of the htmlDependencyReport task executed above, to extract the paths to the JSON files for each sub-project. For example, assuming the output is the following:

    root@d04f6cdbcf81:/src# gradle --init-script init.gradle htmlDependencyReport

> Task :htmlDependencyReport
See the report at: file:///src/build/reports/project/dependencies/index.html

> Task :mail:protocols:pop3:htmlDependencyReport
See the report at: file:///src/mail/protocols/pop3/build/reports/project/dependencies/index.html

> Task :plugins:openpgp-api-lib:openpgp-api:htmlDependencyReport
See the report at: file:///src/plugins/openpgp-api-lib/openpgp-api/build/reports/project/dependencies/index.html

    We should obtain the following list of JSON files for the sub-projects:

    • build/reports/project/dependencies/root.js
    • mail/protocols/pop3/build/reports/project/dependencies/root.mail.protocols.pop3.js
    • plugins/openpgp-api-lib/openpgp-api/build/reports/project/dependencies/root.plugins.openpgp-api-lib.openpgp-api.js

  4. Implement a new file parser in the gemnasium project to parse the dependency list JSON files produced in step 3. above.

    Here's an example of the dependency JSON structure: dependencies.json

    We'll need to make sure we handle the following edge cases:

    • We need to properly parse dependencies that had to undergo conflict resolution, and therefore the requested and selected versions are separated by a right arrow character ➡ (unicode character U+27A1). An example of this situation can be found in this fixture file and the version of org.slf4j/slf4j-api that it resolves to in the gl-dependency-scanning-report.json expectation.

    • Handle the situation where gradle dependencies fails to resolve a dependency, for example:

      $ git clone git@gitlab.com:gitlab-org/security-products/analyzers/gemnasium-gradle-plugin.git && cd gemnasium-gradle-plugin && git checkout 42c5ff41c25fceb561480b379f1886034a09d303

$ docker run -it --rm -e SECURE_LOG_LEVEL=debug -v "$PWD:/gemnasium-gradle-plugin-src" -w /gemnasium-gradle-plugin/src registry.gitlab.com/security-products/gemnasium-maven:2.27.4 bash -ic 'gradle -p /gemnasium-gradle-plugin-src dependencies'

Welcome to Gradle 6.7.1!
...
> Task :dependencies
...
functionalTestImplementationDependenciesMetadata
+--- org.jetbrains.kotlin:kotlin-stdlib:1.5.10
...
+--- org.jetbrains.kotlin:kotlin-test:1.5.10 FAILED
+--- org.jetbrains.kotlin:kotlin-test-junit:1.5.10
|    +--- org.jetbrains.kotlin:kotlin-test:1.5.10 FAILED

      Notice the org.jetbrains.kotlin:kotlin-test:1.5.10 dependency failed to be resolved. We need to make sure to address this situation.

    This step is to be implemented in Implement gemnasium parser for gradle dependenc... (#360626 - closed).

  5. Add unit tests for the new file parser added in step 4. above, similar to the current gradle-dependencies.json test.

  6. Improve image integration test coverage:

    • Add a test with a single dependency with different versions in different scopes (configurations). For example:

      dependencies {
    testRuntimeClasspath 'org.slf4j:slf4j-api:1.7.31'
    implementation 'org.slf4j:slf4j-api:1.7.30'
    runtimeClasspath 'org.slf4j:slf4j-api:1.7.25'
}

    • Add a test to handle gradle dependency constraints as described in this comment. To be more specific, we must be able to show that we can successfully execute a dependency scan against the files in this branch.

    • Add a test to handle the the situation where gemnasium-maven returns the wrong version of a dependency that underwent conflict resolution, as explained in this comment.

    • Add a test to handle the the situation where gradle dependencies fails to resolve a dependency, as described in step 4. above.

    • Add a test with large dependency files, such as the 316 MB file discussed here

    • Add a test against a large open-source gradle project, such as k9mail/k-9. See this comment for more details

  7. Update the Dependency Scanning Documentation to include details explaining that gemnasium-maven uses the htmlDependencyReport task from the Project Report Plugin to generate dependency information.

  8. Remove gemnasium-gradle-plugin parsing code from gemnasium:

/cc @NicoleSchwartz @gonzoyumo @fcatteau

Edited by Adam Cohen