Client-side reports comparison ignores multiple reports

Problem

Some CI reports that are shown in the MR widget use client-side comparison (in the frontend) to determine what changed between the base report and the head report. The problem is that the download path exposed for both base and head reports points to a single artifact and does not consider the scenario where multiple reports of the given type are present in the pipeline.

Lets say I have a pipeline with 2 quality test jobs quality-frontend and quality-backend. The download path for either base or head returns the latest artifact by ID for the given type. In this case it could be either quality-frontend or quality-backend.

The problem is not only limited to code quality reports. At least browser_performance and load_performance are affected too. Maybe more.

Expectation

Both base and head reports should return a combined version of all the reports of the same type. A pipeline can have many reports of the same type and we should fulfill this requirement.

Another option could be to do the comparison in the backend (like it's currently being done for code quality reports) and expose only a single path for frontend to display.

Details

We expose downloadable paths for each report type for a MR. The downloadable_path_for_report_type method uses Ci::Pipeline#batch_lookup_report_artifact_for_file_type which returns "the most recent artifact of a given type" in the pipeline.

This seems to be built under the assumption that a pipeline can only have 1 report of the given type. While this is true for a job, a pipeline could have multiple reports of the same type.

Why are we returning only the most recent one?

Example

Lets say I have a pipeline with 2 quality test jobs quality-frontend and quality-backend. Each job generates its own codeclimate report. When exposing the codeclimate report in the MR we would only return the download path to 1 of them (whatever has the highest ID). This doesn’t seem deterministic, or perhaps I’m not seeing the full picture here. (edited).