Make rate limit configurable for Dependency Scanning SBOM scan API

Release notes

Problem to solve

As determined in the workload analysis and capacity planning, even with caching in place we might still have outlier projects with higher needs.

Inncreasing the default limit is not a sustainable approach and would risk ovewhelming the instance infrastructure if used by too many projects. Instead, we should keep a baseline that addresses most usages but allows outliers to configure particular limits if they need it (and are allowed to).

Proposal

Make the existing API rate limits for Sbom Scan uploads and downloads configurable.

  • dependency_scanning_sbom_scan_api_upload
  • dependency_scanning_sbom_scan_api_download

Telemetry Data Context available in https://gitlab.com/gitlab-org/architecture/readiness/-/merge_requests/35#note_2898032185 (internal)

Rate Limit Configuration Options - GitLab Duo Summary

Option Self-Managed GitLab.com Effort Maintenance Projects Helped Complexity Abuse Risk
1. Rack::Attack ✅ ❌ Low Low All Low Low
2. Instance + Allowlist ⭐ ✅ ✅ Low Minimal All + 2-5 Low Low
3a. Namespace Bounded ✅ ✅ Medium Medium All Medium Medium
3b. Project Allowlist ✅ ✅ Medium Medium All Medium Low
3c. Project Quota ✅ ✅ High High All High Low
4. Project Settings ✅ ⚠️ High High All High High
5. Plan Limits ✅ ✅ Low Low None Low N/A

Recommended Solution: Hybrid Approach - Option 2 (Instance-level) + Manual Allowlist**

For self-managed:

  • ✅ Admins configure instance-wide rate limits
  • ✅ Solves 100% of self-managed cases

For GitLab.com:

  • ✅ Keep default at 400/hour (serves 99%+ of projects)
  • ✅ Manual allowlist for the 2-5 extreme outliers
    • GitLab team manually adds project IDs to allowlist
    • Allowlisted projects get higher limit (e.g., 3000/hour)
    • Requires support ticket + justification
    • Reviewed quarterly

Reasoning:

  • Telemetry shows: P99 usage is 40-55 SBOMs/hour (well within current 400/hour limit)
  • Outliers are minimal: Only 2-5 projects exceed limits even with 80% cache hit rate
  • Self-managed solved: Instance admins can configure limits based on their infrastructure capacity
  • GitLab.com solved: Manual allowlist for the handful of extreme outliers (requires support ticket)
  • Low complexity: Uses existing ApplicationRateLimiter framework, no new UI/API needed for allowlist
  • Pragmatic: Solves 100% of self-managed + 99.99%+ of GitLab.com with minimal engineering effort

Implementation:

  1. Add rate limit settings to application_settings.rate_limits JSONB column
  2. Update existing rate limit definitions to reference settings via Proc
  3. Add Admin UI controls for self-managed configuration
  4. For GitLab.com: Manual project allowlist maintained by GitLab team (internal configuration)
  5. Monitor telemetry; if outliers grow significantly (>20 projects), reconsider namespace-level configuration

When to reconsider:

  • Outliers grow to 20+ projects
  • Pattern emerges (e.g., all large enterprises need this)
  • Self-service becomes important for business reasons
  • Support ticket volume becomes unsustainable

Alternative solution - Option 2 (Instance-level) and Investigate the Outliers

Before implementing complex solutions to support outliers on gitlab.com, understand why these projects need 500-2,700 SBOMs/hour:

Questions to ask:

  1. Are they using the feature correctly?
  2. Is this legitimate usage?
  3. Can we optimize their workflow?
Detailed Analysis of All Options

Rate Limit Configuration Options - Detailed Review

Option 1: Rack::Attack (Instance-level only)

Description: HTTP middleware-level rate limiting

Scope: Instance-level (per user or per IP)

Configuration: application_settings.rate_limits JSONB column

Platform Support:

  • Self-managed: ✅ Admin configurable via UI
  • GitLab.com: ❌ Only GitLab team can configure

Implementation:

  • Extend Gitlab::RackAttack and Gitlab::RackAttack::Request
  • Add settings to Admin area form (app/views/admin/application_settings/_ip_limits.html.haml)
  • Update JSON schema validator
  • Document in User and IP rate limits docs and Application Settings API

Verdict: ❌ Doesn't fit

  • Wrong layer (HTTP middleware, not application logic)
  • No namespace/project-level customization possible
  • SBOM API already uses ApplicationRateLimiter, not Rack::Attack

Option 2: ApplicationRateLimiter with Instance Settings ⭐ RECOMMENDED

Description: Instance-level configuration using existing ApplicationRateLimiter framework

Scope: Instance-level only

Configuration: application_settings fields referenced via Proc in rate_limits hash

Platform Support:

  • Self-managed: ✅ Solves the problem - Admins can configure per-instance limits via Admin UI/API
  • GitLab.com: ⚠️ Only GitLab team can adjust base limits, but can use manual allowlist for outliers

Current Implementation:

# ee/lib/ee/gitlab/application_rate_limiter.rb
dependency_scanning_sbom_scan_api_upload: { threshold: 400, interval: 1.hour }
dependency_scanning_sbom_scan_api_download: { threshold: 6000, interval: 1.hour }

Proposed Implementation:

# ee/lib/ee/gitlab/application_rate_limiter.rb
dependency_scanning_sbom_scan_api_upload: { 
  threshold: -> { 
    if project.id.in?(application_settings.dependency_scanning_sbom_scan_api_project_allowlist || [])
      application_settings.dependency_scanning_sbom_scan_api_upload_limit_allowlist
    else
      application_settings.dependency_scanning_sbom_scan_api_upload_limit
    end
  }, 
  interval: 1.hour 
}

# application_settings (new fields in rate_limits JSONB)
dependency_scanning_sbom_scan_api_upload_limit: 400  # Default
dependency_scanning_sbom_scan_api_upload_limit_allowlist: 3000  # For outliers
dependency_scanning_sbom_scan_api_project_allowlist: [67034029, 59473315]  # Manual list
dependency_scanning_sbom_scan_api_download_limit: 6000

Implementation Steps:

  1. Add new fields to application_settings.rate_limits JSONB column
  2. Update JSON schema validator for rate_limits column
  3. Update rate limit definitions to use Procs referencing settings
  4. Add Admin UI controls in app/views/admin/application_settings/ (for self-managed)
  5. Document in Application Settings API docs
  6. For GitLab.com: GitLab team manually updates allowlist via Rails console/internal tooling

GitLab.com Allowlist Process:

  • User hits rate limit repeatedly
  • Opens support ticket with justification
  • GitLab team reviews usage patterns
  • If legitimate, adds project ID to allowlist
  • Quarterly review of allowlist to remove inactive projects

Effort: Low (1-2 days)

Maintenance: Minimal (quarterly allowlist review)

Verdict: ✅ Best first iteration

  • Simple implementation using proven framework
  • Solves 100% of self-managed cases
  • Solves GitLab.com outliers (2-5 projects) with minimal overhead
  • No complex UI/API needed for allowlist
  • Can iterate to namespace-level config if outliers grow

Option 3a: ApplicationRateLimiter with Bounded Namespace Settings

Description: Namespace-level configuration with hard upper bounds to prevent abuse

Scope: Namespace-level with instance fallback and ceiling enforcement

Configuration: namespace_settings table with validation constraints

Platform Support:

  • Self-managed: ✅ Admins set instance defaults and max ceiling, namespace owners can override within bounds
  • GitLab.com: ✅ Namespace admins can increase limits within GitLab-defined ceiling

Implementation:

Database Schema:

# namespace_settings table (new columns)
dependency_scanning_sbom_scan_api_upload_limit: integer  # Nullable, overrides instance default
dependency_scanning_sbom_scan_api_download_limit: integer

Application Settings:

# application_settings
dependency_scanning_sbom_scan_api_upload_limit: 400  # Default
max_dependency_scanning_sbom_scan_api_upload_limit: 2000  # Ceiling
dependency_scanning_sbom_scan_api_download_limit: 6000
max_dependency_scanning_sbom_scan_api_download_limit: 10000

Validations:

# app/models/namespace_settings.rb
validates :dependency_scanning_sbom_scan_api_upload_limit,
  numericality: { 
    less_than_or_equal_to: -> { Gitlab::CurrentSettings.max_dependency_scanning_sbom_scan_api_upload_limit },
    greater_than: 0,
    allow_nil: true 
  }

Runtime Logic:

def threshold
  namespace_limit = namespace_settings&.dependency_scanning_sbom_scan_api_upload_limit
  instance_default = application_settings.dependency_scanning_sbom_scan_api_upload_limit
  instance_max = application_settings.max_dependency_scanning_sbom_scan_api_upload_limit
  
  [namespace_limit || instance_default, instance_max].min
end

Implementation Steps:

  1. Add columns to namespace_settings table (migration)
  2. Add validations to NamespaceSettings model
  3. Add ceiling settings to application_settings
  4. Create service to read namespace settings and pass runtime threshold
  5. Expose via Groups API with proper authorization
  6. Add Admin UI for namespace admins (optional)
  7. Add audit logging for limit changes
  8. Document in user and admin docs

Safety Guarantees:

  • Hard upper bound per namespace (e.g., max 2000/hour)
  • Instance admin controls the ceiling
  • Model-level validation prevents exceeding ceiling
  • Audit trail for all changes

Risk on GitLab.com:

  • ⚠️ Every namespace could set limits to ceiling
  • ⚠️ Potential for widespread infrastructure impact if many namespaces increase limits
  • ⚠️ Difficult to predict total load

Effort: Medium (2-3 weeks)

Maintenance: Medium (UI, API, validation, monitoring)

Verdict: ⚠️ Overkill for current problem

  • Solves for 2-5 projects with weeks of engineering effort
  • Introduces abuse risk on GitLab.com (unbounded namespace adoption)
  • Better suited if outliers grow to 20+ projects across many namespaces

Option 3b: ApplicationRateLimiter with Project-level Allowlist

Description: Namespace admins can allowlist specific projects for higher limits

Scope: Project-level exceptions to instance limit, managed at namespace level

Configuration: namespace_settings.dependency_scanning_sbom_scan_api_project_allowlist (array of project IDs)

Platform Support:

  • Self-managed: ✅ Admins can allowlist specific projects
  • GitLab.com: ✅ Namespace admins can allowlist up to X projects (e.g., max 10)

Implementation:

Database Schema:

# namespace_settings table (new column)
dependency_scanning_sbom_scan_api_project_allowlist: integer[], default: []

Application Settings:

# application_settings
dependency_scanning_sbom_scan_api_upload_limit: 400  # Default
dependency_scanning_sbom_scan_api_upload_limit_allowlist: 2000  # For allowlisted projects
dependency_scanning_sbom_scan_api_project_allowlist_limit: 10  # Max projects per namespace

Validations:

# app/models/namespace_settings.rb
validates :dependency_scanning_sbom_scan_api_project_allowlist,
  length: { 
    maximum: -> { Gitlab::CurrentSettings.dependency_scanning_sbom_scan_api_project_allowlist_limit }
  }

validate :allowlisted_projects_belong_to_namespace

Runtime Logic:

def threshold
  if project.id.in?(namespace_settings&.dependency_scanning_sbom_scan_api_project_allowlist || [])
    application_settings.dependency_scanning_sbom_scan_api_upload_limit_allowlist
  else
    application_settings.dependency_scanning_sbom_scan_api_upload_limit
  end
end

Safety Guarantees:

  • Limited number of allowlisted projects per namespace (e.g., 10)
  • Two-tier limits: standard (400/hour) vs allowlist (2000/hour)
  • Bounded resource impact: 10 projects × (2000 - 400) = 16,000 additional requests/hour per namespace
  • Easier to audit and monitor than namespace-wide limits

Effort: Medium (2-3 weeks)

Maintenance: Medium (API, validation, monitoring)

Verdict: ⚠️ Better than 3a but still overkill

  • Limits blast radius compared to 3a
  • Still requires significant engineering for 2-5 projects
  • Better suited if outliers are distributed across many namespaces

Option 3c: Per-Project Limits with Namespace Quota

Description: Namespace admins configure custom limits per project, with quota on number of projects

Scope: Project-level limits configured at namespace level, with quota constraints

Configuration: namespace_settings stores per-project limit mappings with quota enforcement

Platform Support:

  • Self-managed: ✅ Admins control quota and ceiling, namespace owners configure projects
  • GitLab.com: ✅ Namespace admins can configure limits for up to X projects

Implementation:

Database Schema:

# namespace_settings table (new column)
dependency_scanning_sbom_scan_api_project_limits: jsonb  # { "project_id": limit }
# Example: { "278964": 1500, "123456": 2000 }

Application Settings:

# application_settings
dependency_scanning_sbom_scan_api_upload_limit: 400  # Default
max_dependency_scanning_sbom_scan_api_upload_limit: 2000  # Ceiling per project
dependency_scanning_sbom_scan_api_project_limit_quota: 10  # Max projects per namespace

Validations:

# app/models/namespace_settings.rb
validates :dependency_scanning_sbom_scan_api_project_limits, 
  length: { 
    maximum: -> { Gitlab::CurrentSettings.dependency_scanning_sbom_scan_api_project_limit_quota }
  }

validate :project_limits_within_ceiling

def project_limits_within_ceiling
  max_allowed = Gitlab::CurrentSettings.max_dependency_scanning_sbom_scan_api_upload_limit
  
  dependency_scanning_sbom_scan_api_project_limits.each do |project_id, limit|
    if limit > max_allowed
      errors.add(:dependency_scanning_sbom_scan_api_project_limits, 
        "cannot exceed maximum of #{max_allowed}")
    end
  end
end

Runtime Logic:

def threshold
  custom_limit = namespace_settings
    &.dependency_scanning_sbom_scan_api_project_limits
    &.dig(project.id.to_s)
  
  custom_limit || application_settings.dependency_scanning_sbom_scan_api_upload_limit
end

Safety Guarantees:

  • Fixed upper bound per project (e.g., 2000/hour)
  • Linear scaling by namespace: Max impact = quota × ceiling per namespace
  • Controlled blast radius: Only X projects per namespace affected
  • Instance admin controls quota, ceiling, and default
  • Audit trail for all changes

Advantages over 3a and 3b:

  • More flexible than 3b (custom limits per project, not just binary allowlist)
  • Safer than 3a (quota limits number of projects affected)
  • Granular control for namespaces with variable needs

Effort: Medium-High (3-4 weeks)

Maintenance: Medium-High (UI, API, validation, monitoring, audit)

Verdict: ⚠️ Most flexible but most complex

  • Highest engineering cost for 2-5 projects
  • Best suited if outliers grow to 50+ projects with varying needs
  • Premature optimization for current problem size

Option 4: ApplicationRateLimiter with Project Settings

Description: Project-level configuration with namespace/instance fallback

Scope: Project-level with cascading from namespace and instance

Configuration: project_settings table + runtime override

Platform Support:

  • Self-managed: ✅ Admins set defaults, project owners can override
  • GitLab.com: ⚠️ Every project could increase limits (high abuse risk)

Implementation: Similar to Option 3a but using project_settings table

Verdict: ❌ Worse than Option 3

  • Higher abuse potential (more projects than namespaces)
  • More granular than needed (SBOM scans already scoped to project)
  • Harder to audit and control

Option 5: Plan Limits

Description: Different limits per subscription tier (Free/Premium/Ultimate)

Scope: Per-tier (Free/Premium/Ultimate)

Configuration: plan_limits table

Platform Support:

  • Self-managed: ✅ Admins can set per-plan limits
  • GitLab.com: ✅ Different limits per tier

Implementation:

# plan_limits table
dependency_scanning_sbom_scan_api_upload_limit: integer
dependency_scanning_sbom_scan_api_download_limit: integer

Verdict: ❌ Doesn't fit

  • Feature is Ultimate-only, no tier differentiation possible
  • Can't solve outlier problem (all outliers are already on Ultimate)
  • Would only work if feature was available across tiers

Implementation plan

Retained solution: Instance-level configuration using existing ApplicationRateLimiter framework without manual allow-list. See #581567 (comment 2913633227)

Implementation Steps:

  1. Add new fields to application_settings.rate_limits by following guidance in https://docs.gitlab.com/development/application_settings/#add-a-new-application-setting
  2. Update ApplicationRateLimiter to use Procs referencing settings: https://gitlab.com/gitlab-org/gitlab/-/blob/28aa596af556aa811b8b77df5fb370228db030b6/ee/lib/ee/gitlab/application_rate_limiter.rb#L41-42
  3. Update Admin settings UI and add a form to update these limits udner the Security and Compliance section
  4. Document in Application Settings API docs
  5. Document in Application Limits
  6. Document in the main Dependency Scanning documentation and any other required place (migration guide?)
  7. ❌ Do not document in Rate Limits. While we effectively use the rate limit framework, this is not a public facing API and from a user perspective this is rather a limit on the DS feature.

Intended users

Feature Usage Metrics

Does this feature require an audit event?

Edited by 🤖 GitLab Bot 🤖