Support multiple reports for client-side comparisons in peformance
This page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.
Problem
Some CI reports that are shown in the MR widget use client-side comparison (in the frontend) to determine what changed between the base report and the head report. The problem is that the download path exposed for both base and head reports points to a single artifact and does not consider the scenario where multiple reports of the given type are present in the pipeline.
Lets say I have a pipeline with 2 quality test jobs quality-frontend and quality-backend. The download path for either base or head returns the latest artifact by ID for the given type. In this case it could be either quality-frontend or quality-backend.
The problem is not only limited to code quality reports. At least browser_performance and load_performance are affected too. The table below reflects which reports need to be changed to support this.
| Report Name | Supports Multiple |
|---|---|
| artifacts:reports:dotenv |
|
| artifacts:reports:performance | |
| artifacts:reports:load_performance | |
| artifacts:reports:browser_performance | |
| artifacts:reports:junit | |
| artifacts:reports:cobertura | |
| artifacts:reports:terraform | |
| artifacts:reports:codequality | MR widget: yes; MR diff annotations: no; Full report: no |
| artifacts:reports:sast | |
| artifacts:reports:secret_detection | |
| artifacts:reports:dependency_scanning | |
| artifacts:reports:container_scanning | |
| artifacts:reports:dast | |
| artifacts:reports:api_fuzzing | |
| artifacts:reports:coverage_fuzzing | |
| artifacts:reports:license_management | Renamed to license_scanning |
| artifacts:reports:license_scanning | |
| artifacts:reports:cluster_image_scanning | |
| artifacts:reports:metrics | |
| artifacts:reports:requirements | |
| artifacts:reports:accessibility |
Expectation
Both base and head reports should return a combined version of all the reports of the same type. A pipeline can have many reports of the same type and we should fulfill this requirement.
Another option could be to do the comparison in the backend (like it's currently being done for code quality reports) and expose only a single path for frontend to display.
Details
We expose downloadable paths for each report type for a MR. The downloadable_path_for_report_type method uses Ci::Pipeline#batch_lookup_report_artifact_for_file_type which returns "the most recent artifact of a given type" in the pipeline.
This seems to be built under the assumption that a pipeline can only have 1 report of the given type. While this is true for a job, a pipeline could have multiple reports of the same type.
Why are we returning only the most recent one?
Example
Lets say I have a pipeline with 2 quality test jobs quality-frontend and quality-backend. Each job generates its own codeclimate report. When exposing the codeclimate report in the MR we would only return the download path to 1 of them (whatever has the highest ID). This doesn’t seem deterministic, or perhaps I’m not seeing the full picture here. (edited).