Skip to content

Component Performance Testing (CPT) - Current State

Context

This issue provide a comprehensive information about CPT including its origin, current state and features.

What is CPT

Component Performance Testing (CPT) is GitLab's automated performance testing framework that runs performance tests in merge request pipelines.

Problem Statement

The problem statement which led to the creation of CPT can be found in CPT blueprint.

Origin and Current state

CPT, as the name suggests, was initially created to performance test standalone components of GitLab architecture like AI Gateway, Gitaly etc. It tested internal api which each component use to communicate with another, as compared to user facing API.

Following few discussions with different teams e.g. 1 , new capability was added to CPT of running tests against a GitLab instance. After adding various capabilities to support that, like data seeding, environment creation of GitLab instance using CNG and reporting, CPT was enabled on GitLab repository.

Following feedback from various teams around the false positives and unreliability of the report, Duo was integrated to

  • Remove outliers and provide more accurate report
  • Analyze Code Diff from MR for any performance change

And based on these two, provide a suggestion if the variability in the tests seems to be environmental or code change related.

Core Features:

  • Can run tests against a component which is a stand alone GitLab component
    • Deploys the component using docker service
    • Tests internal APIs
  • Can run tests against a GitLab CNG instance
    • Deploys GitLab using orchestrator which is an internal tool to deploy CNG instance of GitLab
    • Tests user facing APIs
    • Supports data seeding
    • Support Duo based reporting

What is it Useful For

CPT serves several critical purposes:

Performance Regression Detection:

  • Early Detection: Catches performance regressions before they reach production
  • Merge Request Validation: Provides performance feedback during code review process
  • Performance Insights: Provides performance insights on MRs
  • Non-Blocking: Runs as allow_failure: true to avoid blocking merges

3. Current State

Active Implementation:

  • Pipeline Integration: Fully integrated via e2e:perf-on-cng job in .gitlab/ci/performance-on-cng/main.gitlab-ci.yml
  • Test Coverage: Currently tests two GitLab endpoints i.e. group_merge_request and project_merge_request
  • Reporting: Automated report generation with statistical analysis
  • AI Enhancement: Recently added Duo-powered report generation

Current Test Files Location:

Execution Triggers:

  • Merge request pipelines for gitlab-org namespace projects
  • Master branch pipelines to collect baselines

Current Metrics Collected

CPT currently collects the following metrics for each tests that are run

  • TTFB (Time to First Byte): P90, P95, Mean, Median
  • Response Times: Full request duration metrics
  • Throughput: Requests per second (RPS)
  • Statistical Measures: Standard deviation, coefficient of variation
  • Percentile Analysis: P50, P90, P95 response times

Test Scenarios running currently:

  • Group Merge Requests: Performance of group-level MR endpoints
  • Project Merge Requests: Performance of project-level MR endpoints

Technical Debt and Limitations:

Current Limitations:

  • Single Seed File: Only supports one data seeding file (mr_seed.rb)
  • Limited Baseline Management: Manual baseline updates and management
  • Rule Complexity: Complex CI rules that are difficult to maintain and debug
  • Resource Intensive: Requires significant CI resources for each test run

Proposed Solutions:

  • Modular Seed Files: Support multiple, scenario-specific seed files
  • Automated Baseline Management: Intelligent baseline updates based on stable performance periods
  • Simplified Rule Engine: More intuitive and maintainable CI rule system
  • Resource Optimization: More efficient resource usage and parallel testing capabilities

Related Issues and Documentation

Edited by Vishal Patel