Add an API for on-demand Dependency Scanning using SBOM in the GitLab rails application

Why are we doing this work

To complete Bring security scan results back into the Depen... (&17150) we must update the GitLab Rails application to do the following:

1. Add an API that can receives an SBOM document to be scanned for vulnerabilities. This must only be authorized to be called from a running CI job and must support an asynchronous workflow.
2. Add a new service (in the sense of rails "service object") to execute an on demand Dependency Scanning analysis without tying its results to any state in the vulnerability management system. This must reuse the GitLab SBOM Vulnerability Scanner, so that we maintain a single implementation of our scanner logic.

Relevant links

Non-functional requirements

• Documentation:
• Feature flag:
• Performance:
• Testing:

Implementation plan

Workflow

sequenceDiagram
    participant DS as CI JOB
    participant WH as Workhorse
    participant RAILS as GitLab API
    participant DB as Postgresql DB
    participant BG as Sidekiq Job
    participant FS as File Storage

    Note over DS: CI job starts
    DS->>DS: Dependency Detection & SBOM(s) generation
    loop 
        DS->>WH: Upload SBOM (1 SBOM per query)
        WH->>RAILS: calls `authorize` endpoint
        RAILS->>WH: responds with configuration
        WH->>FS: Store Sbom file
        WH->>RAILS: calls original `upload` endpoint
        RAILS->>DB: Create SbomScan record
        RAILS->>BG: Schedule on-demand DS scan
        RAILS->>WH: Return 202 and download url
        WH->>DS: Return 202 and download url
        DS->>DS: Wait X seconds
        DS->>RAILS: Try to download scan results
        RAILS->>DB: Check SbomScan status
        DB->>RAILS: Status is "created" or  "running"
        RAILS->>DS: Return 202 - "in progress"
        DS->>DS: Wait X seconds   
        Note over DB,FS: On-demand SBOM DS Scan starts
        BG->>DB: Set SbomScan status to "running"
        BG->>FS: Fetch SBOM file
        FS->>BG: Return SBOM file
        BG->>BG: Parse SBOM and perform security analysis
        BG->>FS: Store scan results
        BG->>DB: Set SbomScan status to "finished"
        Note over DB,FS: On-demand SBOM DS Scan completed
        DS->>RAILS: Try to download scan results
        RAILS->>DB: Check SbomScan status
        DB->>RAILS: Status is "finished"
        RAILS->>DS: Return 303 with scan results file url (direct-download)
        DS->>FS: Fetch scan results file
        FS->>DS: Return scan results file
    end
        DS->>DS: Aggregate scan results and generate DS security report
        Note over DS: CI job completed

        Note over DB,FS: Daily cleanup
        BG->>DB: Fetch Expired Sbom Scans (created 2 days ago)
        BG->>FS: Delete stored files
        BG->>DB: Delete SbomScan records

Verification steps

MR breakdown:

1. Introduce SbomScan model and Uploader for DS using SBOM (!195058 (merged))
2. Add services to create and process SbomScan models (!195059 (merged))
3. Add service and worker to destroy expired SbomScan models (!195061 (merged))
4. Add Sbom Scan API endpoints with direct-upload support (!195062 (merged))

These MRs are organized in a stacked diff, thus the complete implementation with all code changes is available in the last MR (!195062 (merged)).