Non-archive artifacts are not available via the Jobs API

Summary

When using the License Compliance feature to generate a license scanning report artifact there appears to be no mechanism to obtain the report artifact via Job Artifacts API. Attempts to get the named report as a single file will 404.

A workaround is to generate the artifact using the paths: keyword, e.g.

license_scanning:
  artifacts:
    paths:
      - gl-license-scanning-report.json

Which requires additional modifications to the pipelines including the License Compliance template, as well as incurring double the storage for the artifact.

Steps to reproduce

Include the License Scanning Template
Attempt to retrieve the report artifact: /api/v4/projects/<project_id>/jobs/<job_id>/artifacts/gl-license-scanning-report.json

The result is a 404.

What is the current bug behavior?

The artifact report is not available via the API.

What is the expected correct behavior?

The artifact report should be available via the API.

Output of checks

This bug happens on GitLab.com and Self-Managed installs both.

Additional Discussion

I'm not certain whether this should be considered a bug or a feature. This was a problem in the UI (and internal API?) when first released - and was discussed and addressed as part of this issue: #218582 (closed) - as well as the related issue about the storage of report artifacts: #23847 (closed)

However, in Issue #218582 (closed) - there was a comment that the 404 was also happening when attempting to get the report artifact via the API. There's an explicit acknowledgement that the work to do the fix likely wouldn't address the API use case and another issue would be needed. As far as I can tell, we didn't create that issue.