Job Artifact API endpoints do not work for report artifacts

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Note

This issue focuses on job-level artifact downloads. Pipeline-level aggregated report downloads are tracked separately in https://gitlab.com/gitlab-org/gitlab/-/issues/582461

Workaround

See https://docs.gitlab.com/api/job_artifacts/#downloading-artifactsreports-files

Customer Impact

See #352644 (comment 2922713306)

Problem to solve

Report artifacts are stored separately (not within the artifacts archive). This causes problems with Job Artifacts API Endpoints that assume the artifacts archive.

Example Report Artifact (and example project here)

artifacts:
  reports:
    dotenv: env

There is an issue (and MR) for making the report artifacts visible via the UI, but the changes were never represented in the API.

The Job API Endpoint uses the build.artifacts_file

desc 'Download a specific file from artifacts archive' do
  detail 'This feature was introduced in GitLab 10.0'
end
...
route_setting :authentication, job_token_allowed: true
get ':id/jobs/:job_id/artifacts/*artifact_path', format: false do
...
  not_found! unless build.available_artifacts?
...
  send_artifacts_entry(build.artifacts_file, path)
end

But this artifacts file will never include the report objects.

# From build.rb
def available_artifacts?
  (!artifacts_expired? || pipeline.artifacts_locked?) && job_artifacts_archive&.exists?
end

def artifacts_file
  job_artifacts_archive&.file
end

User experience goal

I think we either need to mirror UI behavior with what is accessible via the API or provide a more generic endpoint to allow downloading of all of the artifact files.

Prerequisites

Verify that maximum artifact size constraints are properly enforced for report artifacts

Proposal

I don't see an obvious great way to change the current endpoints as they all look to the archive file (which I don't think we do at all(?) with the reports). So I'm suggesting another endpoint that allows general access to any of the job artifacts

# lib/api/ci/job_artifacts.rb
desc 'Download specific artifact from a job'
params do
  requires :job_id, type: Integer, desc: 'The ID of a job'
  requires :artifact_name, type: String, desc: 'Artifact Filename'
end
route_setting :authentication, job_token_allowed: true
get ':id/jobs/:job_id/artifact', urgency: :low do
  authorize_download_artifacts!

  build = find_build!(params[:job_id])
  authorize_read_job_artifacts!(build)

  artifact = build.job_artifacts.find_by file: params[:artifact_name]

  present_carrierwave_file!(artifact&.file)
end

This would allow access to both the archive and the report artifacts without causing breaking changes to the existing archive endpoints. (Probably need a similar endpoint for the pipeline endpoint). Example curl command to grab a sast artifact:

curl -X GET  -H "Private-Token: $TOKEN" "https://gitlab/api/v4/projects/2/jobs/31/artifact?artifact_name=gl-sast-report.json"

Internal Customer Ticket

Edited by Rutvik Shah