### Problem to solve [gemnasium-maven](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven) currently uses the [gemnasium-gradle-plugin](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-gradle-plugin) to generate the dependency list for a project. The `gemnasium-gradle-plugin` presents the following challenges: 1. It's written in `kotlin`, and there are no `kotlin` experts on the Composition Analysis team 1. If changes are made to the plugin, it requires a new release of the plugin, as well as updating `gemnasium-maven` to use the new plugin 1. It's not easy to test the plugin with `gemnasium-maven`, since we need to publish a new version before we can make use of it 1. It requires us to maintain and install a separate non-standard project 1. There are limitations that prevent it from working on all projects Instead of using the `gemnasium-gradle-plugin`, we can leverage the [`htmlDependencyReport`](https://docs.gradle.org/current/dsl/org.gradle.api.reporting.dependencies.HtmlDependencyReportTask.html) task provided by the [Project Report Plugin](https://docs.gradle.org/current/userguide/project_report_plugin.html) to generate the full dependency tree. The `htmlDependencyReport` is part of the standard Gradle distribution, which gives us the following benefits: 1. We don't need to maintain the code ourselves 1. It ships with Gradle, so we don't need to package and install a separate non-standard project 1. It produces JSON output 1. It supports more projects than the `gemnasium-gradle-plugin` Because of these advantages, we should replace the `gemnasium-gradle-plugin` with the `htmlDependencyReport` task, which is the purpose of this issue. Follow-up: run the `androidDependencies` task to implement Android support. See https://gitlab.com/gitlab-org/gitlab/-/issues/336866 ### Proposal Update [`gemnasium-maven`](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven) so that it uses the [`htmlDependencyReport`](https://docs.gradle.org/current/dsl/org.gradle.api.reporting.dependencies.HtmlDependencyReportTask.html) task to generate the dependency tree. This will require us to update the [mvnplugin.Parse](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium/-/blob/4971bff33290766838ed479d604fa9f8783de740/scanner/parser/mvnplugin/mvnplugin.go#L17-35) function to handle the `JSON` output by the `htmlDependencyReport` task. ### Further details JSON files are generated by the [JsonProjectDependencyRenderer](https://github.com/gradle/gradle/blob/8b44028386a1c23105fdd2f9b8a4eb86156e09a3/subprojects/diagnostics/src/main/java/org/gradle/api/reporting/dependencies/internal/JsonProjectDependencyRenderer.java#L114) class. ### Intended users * [Sasha (Software Developer)](https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#sasha-software-developer) * [Simone (Software Engineer in Test)](https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#simone-software-engineer-in-test) ### User experience goal Projects are analyzed for dependencies the same way they currently are, but we support more projects. #### Current implementation Right now `gemnasium-maven` has a [`gradle` builder](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/-/blob/v2.23.0/builder/gradle/gradle.go) that [runs the `gemnasiumDependenciesTask`](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/-/blob/v2.23.0/builder/gradle/gradle.go#L80) handled by the [Gemnasium Maven Plugin](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven-plugin). The task creates a JSON file that lists the dependencies of all Gradle dependency configurations, and this file is parsed by the [mvnplugin file parser](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium/-/blob/v2.29.9/scanner/parser/mvnplugin/mvnplugin.go#L29). The parser simply collects the package names and versions; it doesn't return a dependency graph. #### `htmlDependencyReport` task The [`htmlDependencyReport`](https://docs.gradle.org/current/dsl/org.gradle.api.reporting.dependencies.HtmlDependencyReportTask.html) task provided by the [Project Report Plugin](https://docs.gradle.org/current/userguide/project_report_plugin.html) is shipped with the Gradle source. In order to make use of it, we need to add the `project-report` plugin by creating a [Gradle Initialization Script](https://docs.gradle.org/current/userguide/init_scripts.html) with the following contents: ```shell allprojects { apply plugin: 'project-report' } ``` Assuming we've placed the above commands into a file named `init.gradle`, we can then run the `htmlDependencyreport` task as follows: ```shell gradle --init-script init.gradle htmlDependencyReport ``` This command will output a `js` file for each sub-project detected within the given project, and produces the following output: ```shell > Task :htmlDependencyReport See the report at: file:///src/build/reports/project/dependencies/index.html > Task :mail:protocols:pop3:htmlDependencyReport See the report at: file:///src/mail/protocols/pop3/build/reports/project/dependencies/index.html > Task :plugins:openpgp-api-lib:openpgp-api:htmlDependencyReport See the report at: file:///src/plugins/openpgp-api-lib/openpgp-api/build/reports/project/dependencies/index.html ``` The `js` files for the above sub-projects are located in the following files: - `build/reports/project/dependencies/root.js` - `mail/protocols/pop3/build/reports/project/dependencies/root.mail.protocols.pop3.js` - `plugins/openpgp-api-lib/openpgp-api/build/reports/project/dependencies/root.plugins.openpgp-api-lib.openpgp-api.js` We can convert the `Task :<sub-project-name>:htmlDependencyReport` string from the above text output into a `js` file path as follows: ```go func taskStringToPath(taskStr string) string { taskStr = strings.TrimPrefix(taskStr, "Task :") taskStr = strings.TrimRight(taskStr, ":htmlDependencyReport") fileName := strings.Replace(taskStr, ":", ".", -1) filePrefix := strings.Replace(taskStr, ":", "/", -1) return fmt.Sprintf("%s/build/reports/project/dependencies/root.%s.js", filePrefix, fileName) } ``` This `js` file has the following structure and contains the dependency tree in `JSON` format: ``` var projectDependencyReport = {"gradleVersion":"Gradle 7.4","generationDate":"Wed Mar 30 06:48:21 UTC 2022","project":{"name":"plugins","description":null,"configurations":[{"name":"ktlint","description":"Main ktlint-gradle configuration","dependencies":[{"module":"com.pinterest:ktlint","name":"com.pinterest:ktlint:0.40.0","resolvable":"RESOLVED","hasConflict":false}]}]}}; ``` If we remove the JavaScript preamble `var projectDependencyReport = ` from the above `js` file, as well as the trailing semicolon `;`, then we end up with valid `JSON`. See [here](https://gitlab.com/gitlab-org/security-products/tests/java-gradle/-/blob/3053bc86b608b5da45d945a2f071e81c7cac7a20/.gitlab-ci.yml) for an example of running the `htmlDependencyReport` task and the resulting JSON is available [here](https://gitlab-org.gitlab.io/-/security-products/tests/java-gradle/-/jobs/2181213450/artifacts/dependencies.json). By default, the `htmlDependencyReport` task lists all dependencies of all configurations (scopes), which means that we'll need to de-duplicate these entries. See [Listing dependencies in a project](https://docs.gradle.org/current/userguide/viewing_debugging_dependencies.html#sec:listing_dependencies): > Every Gradle project provides the task dependencies to render the so-called dependency report from the command line. By default the dependency report renders dependencies for all configurations. #### Multi-project build In a multi-project Gradle build, the `htmlDependencyReport` task lists the dependencies of all sub-projects. ### Documentation Add documentation to explain that `gemnasium-maven` uses the [`htmlDependencyReport`](https://docs.gradle.org/current/dsl/org.gradle.api.reporting.dependencies.HtmlDependencyReportTask.html) task provided by the [Project Report Plugin](https://docs.gradle.org/current/userguide/project_report_plugin.html) to generate the dependency tree. ### Testing `gemnasium-maven` has [multiple image tests](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/blob/bebc810/spec/image_spec.rb#L229-363) for Gradle. The new feature added by this issue must not break any of these existing tests. ### What does success look like, and how can we measure that? A dependency list can be generated using the [`htmlDependencyReport`](https://docs.gradle.org/current/dsl/org.gradle.api.reporting.dependencies.HtmlDependencyReportTask.html) task without breaking existing tests. We also believe this will resolve our android issues - and should validate asap (while in testing, pre-release) with @Regis.Guillermin and any other customers we become aware of who are using android and willing to test an image. (do not wait on customer feedback if this works for the test cases however). See https://gitlab.com/gitlab-org/gitlab/-/issues/336866 ### What is the type of buyer? gitlab~11736296 ~"GitLab Ultimate" ### Is this a cross-stage feature? No, this only affects gitlab~10690742 ### Implementation plan 1. [x] https://gitlab.com/gitlab-org/gitlab/-/issues/437268+s 1. [x] https://gitlab.com/gitlab-org/gitlab/-/issues/439450+s 1. [x] https://gitlab.com/gitlab-org/gitlab/-/issues/360626+s 1. [x] https://gitlab.com/gitlab-org/gitlab/-/issues/444203+s 1. [x] https://gitlab.com/gitlab-org/gitlab/-/issues/446145+s 1. [x] https://gitlab.com/gitlab-org/gitlab/-/issues/437803+s 1. [x] https://gitlab.com/gitlab-org/gitlab/-/issues/451435+s 1. [x] https://gitlab.com/gitlab-org/gitlab/-/issues/437275+s 1. [x] https://gitlab.com/gitlab-org/gitlab/-/issues/443848+s 1. [x] https://gitlab.com/gitlab-org/gitlab/-/issues/443847+s 1. [x] https://gitlab.com/gitlab-org/gitlab/-/issues/458370+s 1. [x] https://gitlab.com/gitlab-org/gitlab/-/issues/437277+s 1. [x] https://gitlab.com/gitlab-org/gitlab/-/issues/446088+s 1. [x] https://gitlab.com/gitlab-org/gitlab/-/issues/437278+s 1. [X] https://gitlab.com/gitlab-org/gitlab/-/issues/462930+s 2. [x] https://gitlab.com/gitlab-org/gitlab/-/issues/462792+s ### Engineering DRI @philipcunningham <hr/> /cc @NicoleSchwartz @gonzoyumo @fcatteau