Add ability to download reports from Secure jobs

We have an API for downloading artifacts on a per job basis.

See:

https://docs.gitlab.com/ee/api/job_artifacts.html

Need

There is an increasing need to allow for artifact downloads of a particular job type when it pertains to devopssecure jobs. In particular Fuzzing and SAST. Several issues have requested the need to have a single URL to hit that contains a file download for the entire pipeline's artifacts aggregated from all jobs that contain a certain report_type.

Requirements

  • Pipeline API endpoint that allows for downloading a single file of all artifacts in a pipeline for a specific set of job that contain a user provided list of report_types. We have been using the stage: 'fuzz' field from GET /projects/:id/jobs/:job_id/artifacts,
stage: 'fuzz'

This is fragile. We should move over to using report_type in our new implementation.

Endpoints to implement in graphQL:

  1. Get list of all jobs in a pipeline that contain at least one of the report types defined in user provided "report_types" (Should only need pipeline ID and report_types and scope it by project id?)

  2. GraphQL endpoint for downloading artifacts for individual jobs. (We only need a job ID to find this, can be scoped by project ID as well)

Not in scope for this issue

  1. GraphQl endpoint for downloading aggregated job artifacts in a pipeline that contain at least one of the report types defined in user provided "report_types" (Should only need pipeline ID and report_types and scope it by project id?)

File Naming

  • For the job artifact and pipeline artifact downloads, we should improve the file naming. Right now its just artifacts.zip

  • For job artifacts maybe something like <JOB_NAME>-<JOB_ID>.zip

  • For pipeline artifacts maybe something like <PIPELINE_NAME>-<PIPELINE_ID>.zip with sub folders inside of the format <JOB_NAME>-<JOB_ID>.zip

Support other report types other than just Secure

  • Attempt to make this work for all report types not just Security related report types.

Example Screenshots:

Fuzzing

Issue: #210343 (closed)

Issue: #217151 (closed)

Pipeline_list

SAST

Epic: &4388

Technical details

Backend

The frontend needs the following GraphQL query:

{
  project(fullPath: "project-full-path") {
    pipeline(iid: "the-iid") {
      jobs(securityReportTypes: [FUZZING, SAST]) {
        nodes {
          artifacts(types: [ARCHIVE]) {
            nodes {
              downloadPath
            }
          }
        }
      }
      # reportsArchive(reportTypes: [FUZZING, SAST]) {  <-- deferred to a future issue
        # downloadPath
      # }
    }
  }
}

The following new GraphQL fields are needed:

  1. EE::Types::Ci::PipelineType.jobs that takes a securityReportTypes argument and returns all jobs with a report key that matches. Valid report types are any value from Security::SecurityJobsFinder.allowed_job_types, which should be created as an enum type JobArtifactSecurityReportTypesEnum so the frontend knows what values are valid
  2. EE::Types::Ci::JobType.artifacts that takes a types argument and returns artifacts of the given types. Valid values for the types argument are any value from Ci::JobArtifact::TYPE_AND_FORMAT_PAIRS, which should be created as an enum type JobArtifactTypesEnum so the frontend knows what values are valid
  3. A new EE::Types::Ci::JobArtifactType that has a downloadPath field that returns the path needed to download the artifact and a fileType field that returns the file type of the download artifact
  4. Types::Ci::PipelineType.reportsArchive which takes a reportTypes filter and is a new Types::Ci::ReportsArchive type that has a downloadPath. That path will need to route to a new action that fetches the reports matching the given report types and zips them into an archive for download (Deferred to a future issue)

Notes

  • Currently, Security::JobsFinder can be used to fetch jobs from a pipeline that have a given Secure report type. This should be merged into Ci::JobsFinder so that Types::Ci::JobType.artifacts(reportType: []) can be used by other stages This plan has been abandoned. See #251015 (closed)
  • Each of these fields should be resolved in such a way that no N+1 queries are created. LooksAhead might be helpful

Frontend

TBD

Edited by rossfuhrman