Performance Testing for Vulnerabilities Across Branches on GitLab Dedicated-type Deployment

Summary

We need assistance with performance testing for the Vulnerabilities Across Branches feature on a GitLab Dedicated-type deployment (or self-managed instance using reference architecture) to validate stability and performance before the closed beta rollout.

Background

The Vulnerabilities Across Branches feature (tracked in https://gitlab.com/gitlab-org/gitlab/-/issues/571797) is planned for closed beta rollout at the end of FY2025Q1 (January). This feature allows users to track vulnerability information across multiple branches beyond just the default branch.

Due to significant stability and performance risks (detailed in the parent issue), we need to validate the feature's behavior on a GitLab Dedicated-type deployment before broader rollout.

Key Risks to Test

  1. Database Growth and Query Performance: Vulnerability tables could grow substantially (potentially 16x current size with full utilization), which may impact query performance and database maintenance operations
  2. Write Node Pressure: Increased vulnerability ingestion from multiple branches could saturate write nodes
  3. Sidekiq Worker Contention: Increased security scan processing demand across multiple workers and queues
  4. Resource Contention: Non-.com instances lack the decomposed Sec DB setup, making them more vulnerable to resource contention

Testing Requirements

We need to:

  1. Set up a GitLab instance using reference architecture (on sandbox cloud) that represents a common Dedicated deployment
  2. Set up test data that simulates realistic vulnerability data volumes
  3. Run E2E tests to validate functionality
  4. Run performance tests focusing on:
    • Database query performance under increased vulnerability data load
    • Write node CPU and WAL processing capacity
    • Sidekiq worker queue processing times
    • Overall system stability metrics

Specific Performance Metrics to Monitor

  • Database query execution times for vulnerability reports
  • Database table sizes and growth rates
  • Write node CPU saturation and WAL processing
  • Read replica lag
  • Sidekiq queue latency for security-related workers (extensive list available in parent issue)
  • Pipeline processing times for security scans

Expected Outcomes

  • Validation that a reference architecture deployment can handle the increased load from tracking vulnerabilities across multiple branches
  • Identification of any performance bottlenecks or stability issues
  • Baseline metrics for comparison with GitLab.com deployment
  • Recommendations for deployment sizing or configuration adjustments if needed

Timeline

Testing should be completed before the closed beta rollout planned for end of FY2025Q1 (January 2026).

Related Issues

Additional Context

GitLab.com has a decomposed Sec DB which provides some isolation, but Dedicated and self-managed instances do not. This makes performance testing on a Dedicated-type deployment critical to ensure we don't introduce stability issues for these customers.

cc @ghavenga @hmuralidhar @mclausen35