Add ability to download reports from Secure jobs
We have an API for downloading artifacts on a per job basis.
See:
https://docs.gitlab.com/ee/api/job_artifacts.html
Need
There is an increasing need to allow for artifact downloads of a particular job type when it pertains to devopssecure jobs. In particular Fuzzing and SAST. Several issues have requested the need to have a single URL to hit that contains a file download for the entire pipeline's artifacts aggregated from all jobs that contain a certain report_type
.
Requirements
- Pipeline API endpoint that allows for downloading a single file of all artifacts in a pipeline for a specific set of job that contain a user provided list of
report_types
. We have been using thestage: 'fuzz'
field fromGET /projects/:id/jobs/:job_id/artifacts
,
stage: 'fuzz'
This is fragile. We should move over to using report_type
in our new implementation.
Endpoints to implement in graphQL:
-
Get list of all jobs in a pipeline that contain at least one of the report types defined in user provided "report_types" (Should only need pipeline ID and report_types and scope it by project id?)
-
GraphQL endpoint for downloading artifacts for individual jobs. (We only need a job ID to find this, can be scoped by project ID as well)
Not in scope for this issue
- GraphQl endpoint for downloading aggregated job artifacts in a pipeline that contain at least one of the report types defined in user provided "report_types" (Should only need pipeline ID and report_types and scope it by project id?)
File Naming
-
For the job artifact and pipeline artifact downloads, we should improve the file naming. Right now its just
artifacts.zip
-
For job artifacts maybe something like
<JOB_NAME>-<JOB_ID>.zip
-
For pipeline artifacts maybe something like
<PIPELINE_NAME>-<PIPELINE_ID>.zip
with sub folders inside of the format<JOB_NAME>-<JOB_ID>.zip
Support other report types other than just Secure
- Attempt to make this work for all report types not just Security related report types.
Example Screenshots:
Fuzzing
Issue: #210343 (closed)
Issue: #217151 (closed)
SAST
Epic: &4388
Technical details
Backend
The frontend needs the following GraphQL query:
{
project(fullPath: "project-full-path") {
pipeline(iid: "the-iid") {
jobs(securityReportTypes: [FUZZING, SAST]) {
nodes {
artifacts(types: [ARCHIVE]) {
nodes {
downloadPath
}
}
}
}
# reportsArchive(reportTypes: [FUZZING, SAST]) { <-- deferred to a future issue
# downloadPath
# }
}
}
}
The following new GraphQL fields are needed:
-
EE::Types::Ci::PipelineType.jobs
that takes asecurityReportTypes
argument and returns all jobs with a report key that matches. Valid report types are any value fromSecurity::SecurityJobsFinder.allowed_job_types
, which should be created as an enum typeJobArtifactSecurityReportTypesEnum
so the frontend knows what values are valid -
EE::Types::Ci::JobType.artifacts
that takes atypes
argument and returns artifacts of the given types. Valid values for thetypes
argument are any value fromCi::JobArtifact::TYPE_AND_FORMAT_PAIRS
, which should be created as an enum typeJobArtifactTypesEnum
so the frontend knows what values are valid - A new
EE::Types::Ci::JobArtifactType
that has adownloadPath
field that returns the path needed to download the artifact and afileType
field that returns the file type of the download artifact -
(Deferred to a future issue)Types::Ci::PipelineType.reportsArchive
which takes areportTypes
filter and is a newTypes::Ci::ReportsArchive
type that has adownloadPath
. That path will need to route to a new action that fetches the reports matching the given report types and zips them into an archive for download
Notes
-
Currently,This plan has been abandoned. See #251015 (closed)Security::JobsFinder
can be used to fetch jobs from a pipeline that have a given Secure report type. This should be merged intoCi::JobsFinder
so thatTypes::Ci::JobType.artifacts(reportType: [])
can be used by other stages - Each of these fields should be resolved in such a way that no N+1 queries are created. LooksAhead might be helpful
Frontend
TBD