Add soft rate limit to SBOM vulnerability scans

What does this MR do and why?

Background

With Add an API for on-demand Dependency Scanning us... (#542831 - closed) we added the capability to scan an SBOM document for vulnerabilities in the GitLab rails backend (not released yet, it's behind a Feature flag, disabled by default). Since this process uses Sidekiq background jobs rather than CI jobs, it requires additional considerations to handle scalability and limits.

The initial implementation included some basic application rate limits as a temporary measure, but it was clear that a more thorough approach was required before launching the feature.

⚠️ This feature is behind the dependency_scanning_sbom_scan_api feature flag that is disabled by default.

What

This MRs introduces a soft rate limit mechanism to the SBOM vulnerability scans API, scoped by project.

Normal usage (under threshold):

• Scans processed on high-priority worker (ProcessSbomScanWorker)
• Fast processing with :high urgency
• Standard API responses

Heavy usage (over threshold):

• Scans routed to throttled worker (ProcessSbomScanThrottledWorker)
• Lower urgency (:low) with higher concurrency limits
• API returns additional properties in response body to inform client
• Client displays a warning about increased processing time

Why

• Maintains availability: Heavy users can still use the feature beyond the soft limit
• Preserves service quality: Prevents resource contention affecting other users
• Transparent communication: Clients are informed about throttling via properties n response body
• Gradual degradation: Performance degrades gracefully rather than failing

Implementation changes:

• Add a ProcessSbomScanThrottledWorker that does the same job as ProcessSbomScanWorker but using a lower urgency and higher concurrency
• Add a dependency_scanning_sbom_scan_api_throttling application limit and use it in SBOM upload API endpoint
• Route scans requests above threshold to the newly introduced throttled worker
• Update the SBOM Scan API response to return additional properrties when scan are throttled
• Maintain hard application limits on the API endpoints with higher values, to prevent runaway usage

Performance analysis

See private note: !204070 (comment 2736964026)

References

• Implement soft rate limiting for SBOM Scan Proc... (#561759 - closed)

Screenshots or screen recordings

Before	After

How to set up and validate locally

1. Follow the steps to enable the API on your GDK: !195062 (merged)
2. Manually adjust :dependency_scanning_sbom_scan_api_throttling threshold in the ApplicationRateLimiter to use a lower value for testing purposes
3. Trigger multiple CI jobs to reach the limit and verify that:
   1. the throttled worker is used gdk tail rails-background-jobs | grep SbomScan
   2. the response provide the throttled and project_throttling_resets_in properties

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.