Enhance CI schema validation to prevent specification of multiple report files for reports which only support a single file (e.g. codequality)

This issue proposes enhancements to CI pipeline configuration validation, documentation and error reporting to prevent user confusion over the cause of job failures due to the inadvertent use of unsupported report artifact file matching specifications.

Proposal

GitLab CI/CD supports different types of artifacts:reports that are used as sources of information displayed in security dashboards, the MR load testing widget, project requirements, MR Terraform widget etc.

Looking at the codequality report artifact type, source files can be selected by specifying one or more source files as an array or one or more values or a file glob pattern, e.g.:

artifacts:
  reports:
    codequality:
      - cq-file1.json

artifacts:
  reports:
    codequality:
      - cq-file1.json
      - cq-file2.json

artifacts:
  reports:
    codequality: [ cq-file1.json, cq-file2.json ]

artifacts:
  reports:
    codequality:
      - cq-file*.json

However, if more than one file matches the specification the job will fail in the Uploading artifacts... stage with the following error:

ERROR: Uploading artifacts as "codequality" to coordinator... error  error=couldn't execute POST against https://gitlab.example.com/api/v4/jobs/1600/artifacts?artifact_format=raw&artifact_type=codequality: Post "https://gitlab.example.com/api/v4/jobs/1600/artifacts?artifact_format=raw&artifact_type=codequality": only one file can be sent as raw id=1600 token=64_xssZK

This error occurs because the codequality artifact file format is set to raw in app/models/ci/job_artifacts.rb.

Some report types such as cyclonedx do allow multiple files to be uploaded as report artifacts, and these have a file format of gzip.

Ideally all report types would support the specification of multiple files which are combined as required by the backend/frontend to allow them to be processed and displayed correctly, but while only some report types do this it would be helpful if the pipeline configuration schema validation was enhanced to prevent the user being able to specify multiple files that will cause the job to fail.

(As a side note the tooltip message displayed when you hover over codequality in the Pipeline Editor reads Path to file or list of files with code quality report(s) (such as Code Climate). which is misleading.)

The report types that are currently set to raw format and which may benefit from similar enhancements to the CI schema to prevent unsupported file selection specifications being used are:

      # Security reports and license scanning reports are raw artifacts
      # because they used to be fetched by the frontend, but this is not the case anymore.
      sast: :raw,
      secret_detection: :raw,
      dependency_scanning: :raw,
      container_scanning: :raw,
      cluster_image_scanning: :raw,
      dast: :raw,
      license_scanning: :raw,

      # All these file formats use `raw` as we need to store them uncompressed
      # for Frontend to fetch the files and do analysis
      # When they will be only used by backend, they can be `gzipped`.
      accessibility: :raw,
      codequality: :raw,
      performance: :raw,
      browser_performance: :raw,
      load_performance: :raw,
      terraform: :raw,
      requirements: :raw,
      requirements_v2: :raw,
      coverage_fuzzing: :raw,
      api_fuzzing: :raw

Alternative (or Complementary) Proposal

If the CI schema can't be enhanced in this way, an alternative approach to help prevent "mis-configured" jobs and user confusion would be to:

• add a Job Artifacts Troubleshooting item noting the above report artifacts upload error and the likely cause being the selection of multiple report files where only one is supported
• update the GitLab CI/CD artifacts reports types documentation to indicate for each raw format report type than only one file should be configured
• include in the docs examples of how to combine multiple report files into a single file where required
• enhance the error reporting of the runner helper to explicitly indicate when the artifact upload of a report artifact has failed due to >1 file being matched e.g.

ERROR: Uploading artifacts as "codequality" to coordinator. Only a single file can be used by the `codequality` report artifact configuration.