Add Sbom Scan API endpoints with direct-upload support

Overall goal

This is a new approach to our Dependency Scanning feature that reuses the GitLab Sbom Vulnerability Scanner developed in the rails platform for scanning SBOM files generated in a CI job. This allows GitLab to centralize the Dependency Scanning feature on a single scanning engine for various scanning contexts. Please see the epic Bring security scan results back into the Depen... (&17150 - closed) for more details.

This MR is part of a stacked diff:

1. Introduce SbomScan model and Uploader for DS using SBOM (!195058 (merged))
2. Add services to create and process SbomScan models (!195059 (merged))
3. Add service and worker to destroy expired SbomScan models (!195061 (merged))
4. ➡️ Add Sbom Scan API endpoints with direct-upload support (!195062 (merged))

Thus, the complete implementation with all code changes is available in the last MR (!195062 (merged)).

Notes:

• This feature will be released behind the feature flag dependency_scanning_sbom_scan_api included in the last MR (with the "user facing" API).
• These SBOM scans are only meant to be ephemeral and thus they don't interact with the Vulnerability Management system.

Video recordings

• Introduction and workflow: https://youtu.be/kjlzwhLKZxQ
• MRs walkthrough: https://youtu.be/vFLvOUZOMxA

What does this MR do and why?

Summary

This merge request adds a new API for handling SBOM (Software Bill of Materials) vulnerability scanning in GitLab's CI/CD system. The changes create three API endpoints that allow CI jobs to upload SBOM files for security scanning, check the authorization for uploads (direct-upload), and download the scan results once processing is complete.

The implementation includes proper authentication using job tokens (similar to CI job artifacts), file upload handling with support for both local and remote storage, and different response codes based on the scan status (queued, in progress, completed, or failed). A new feature flag called dependency_scanning_sbom_scan_api controls access to this functionality, allowing it to be gradually rolled out.

Comprehensive tests ensure the API works correctly for various scenarios including successful uploads, error conditions, and different storage configurations.

Key Features Added

Three New API Endpoints:

1. POST /jobs/:id/sbom_scans/authorize - Authorizes SBOM file uploads (direct-upload support)
2. POST /jobs/:id/sbom_scans - Uploads SBOM files for scanning
3. GET /jobs/:id/sbom_scans/:sbom_scan_id/download - Downloads scan results

Rate Limiting Implementation:

• Upload rate limit: 50 requests/hour per project
• Download rate limit: 150 requests/hour per project (3x upload rate matching the client's 3 attempts when polling)
• Scoped to project level for proper isolation

Note: A more comprehensive rate limiting system will be implemented in a follow-up issue, before the global rollout of the feature: Implement soft rate limiting for SBOM Scan Proc... (#561759 - closed)

Storage & Upload Support:

• Support local and remote storage
• Workhorse integration
• Direct upload support with final path storage optimization

Authentication & Authorization:

• Job token authentication (similar to CI artifacts)
• License validation (requires dependency_scanning feature available, Ultimate license)
• Feature flag protection for controlled rollout
• Project-scoped access control

Status-Based Download Responses:

• 202 Accepted: Scan queued (created) or in progress (running)
• 200 OK: Scan completed (finished) - returns JSON results
• 410 Gone: Scan failed - includes error message
• 500 Internal Server Error: Unexpected status
• 404 Scan record not found

Technical Implementation Details

Model Changes:

• Added for_build scope to SbomScan model for proper job isolation
• Enhanced uploader with direct-upload path generation

Workhorse Integration:

• Added route pattern: /api/v4/jobs/[0-9]+/sbom_scans
• Integrated with multipart upload handling
• Updated upload test coverage

Security & Compliance:

• Proper job token validation prevents cross-job access
• Project-level scoping ensures isolation
• Rate limiting prevents abuse
• License enforcement for enterprise features

References

• Issue: Add an API for on-demand Dependency Scanning us... (#542831 - closed)
• Epic: Bring security scan results back into the Depen... (&17150 - closed)

Screenshots or screen recordings

Before	After

How to set up and validate locally

1. checkout the branch and run the DB migration

   git checkout ogonzalez-Add-DS-Sbom-Scan-API-01eb42f1
   bin/rails db:migrate:up:main RAILS_ENV=development --trace

2. Make sure Object Storage is enabled in your GDK.

3. Make sure that your local copy of workhorse in your GDK is rebuilt with the necessary route:

   cd workhorse && make
   gdk restart gitlab-workhorse

4. Make sure to Enable PMDB synchronization features on your GDK and sync advisories.

5. Clone this example project in your GDK instance: https://gitlab.com/gitlab-org/secure/tests/olivier/on-demand-ds-scan-api/-/tree/main

6. Enable the feature flag in a rails console for this project : Feature.enable(:dependency_scanning_sbom_scan_api, project)

7. Run a pipeline on the main branch. This will execute a dummy dependency_scanning CI job that:

   • uploads the sbom_file.json file present in the repo
   • extract the download_url from the response
   • calls the download url after a couple of seconds
   • display the content of the scan results and save them to an artifact

You can also open the minio admin console (minio/gdk-minio) and browse Monitoring -> Watch. Select the uplloads bucket and click the Start button. Here you will see all the files uploaded and fetched on the object storage backend. When running the above pipeline, two files should be added to the uploads bucket.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.