Replace deprecated quarantine tracking with ClickHouse-based reports

David Dieulivolrequested to merge
replace-quarantine-with-clickhouse into master

Context

Closes #1720

Depends on gitlab-org/ruby/gems/gitlab-triage!366 (merged) (merged and released in gitlab-triage 1.48.0)

ClickHouse is now the SSOT for quarantined tests. The old approach of tracking quarantine issues via GitLab labels is deprecated and incomplete.

What's in this MR?

Removes deprecated quarantine tracking:

  • quarantined_broken_specs_summary and quarantined_flaky_specs_summary methods
  • flaky_tests_heatmap from all team summary reports (uses deprecated quarantine label data)
  • Related conditions from policy templates

Adds ClickHouse-based quarantine reporting:

  • New quarantined_tests(group_key:) helper in lib/quarantine_helper_context.rb
  • Uses ClickhouseQuarantineHelper to fetch data directly from ClickHouse
  • Data comes with accurate, real-time information
  • Includes links to quarantine issues and Grafana dashboards
  • Appears as custom type summary in team reports

Security cleanup:

  • Remove .env file from repository (should never be committed)
  • .env.example remains as template for local configuration
  • Updated config/boot.rb to safely load .env in local development only (never in CI or production)

Dependencies:

  • Upgraded to gitlab-triage 1.48.0 (required for custom type summary support)

Proof of work

Local testing instructions

Prerequisites:

  1. ClickHouse credentials (get from 1Password or CI variables)
  2. gitlab-triage 1.48.0 or higher

Setup:

# 1. Copy the example file
cp .env.example .env

# 2. Edit .env and fill in your ClickHouse credentials:
# CLICKHOUSE_URL=https://your-clickhouse-url
# CLICKHOUSE_DATABASE=test_metrics
# CLICKHOUSE_USERNAME=your_username
# CLICKHOUSE_PASSWORD=your_password

# 3. Verify gitlab-triage version
bundle exec gitlab-triage --version
# Should show: 1.48.0

Run dry-run for a team:

bundle exec gitlab-triage --dry-run \
  -r ./plugins/all \
  --token $GITLAB_API_PRIVATE_TOKEN \
  --source-id gitlab-org/gitlab \
  --policies-file policies/generated/team-summary.yml.erb/pipeline_execution.yml

Expected output: The quarantine section should show data fetched from ClickHouse (e.g., "59 quarantined tests").

CI dry-run output

Job: https://gitlab.com/gitlab-org/quality/triage-ops/-/jobs/13019125376

Result: Successfully fetched 59 quarantined tests from ClickHouse

See triage report

Quarantined tests

59 quarantined tests - View dashboard

View quarantined tests

Important: We started to import quarantine data in ClickHouse on 2026-01-08 for E2E tests, and on 2026-01-19 for non-E2E tests. Quarantined Since will not go further than that date.

Test Case Quarantined Since Issue
qa/specs/features/browser_ui/4_verify/ci_job_artifacts/unlocking_job_artifacts_across_parent_child_pipelines_spec.rb:178 2026-01-08 (4 weeks) View Issue
qa/specs/features/ee/browser_ui/4_verify/pipeline_for_project_mirror_github_spec.rb:55 2026-01-08 (4 weeks) View Issue
qa/specs/features/browser_ui/4_verify/ci_job_artifacts/unlocking_job_artifacts_across_parent_child_pipelines_spec.rb:153 2026-01-08 (4 weeks) View Issue
qa/specs/features/ee/browser_ui/4_verify/merge_train_with_multiple_cars_spec.rb:45 2026-01-08 (4 weeks) View Issue
qa/specs/features/browser_ui/4_verify/ci_job_artifacts/unlocking_job_artifacts_across_parent_child_pipelines_spec.rb:128 2026-01-08 (4 weeks) View Issue
qa/specs/features/ee/browser_ui/4_verify/pipeline_for_merged_result_spec.rb:56 2026-01-08 (4 weeks) View Issue
qa/specs/features/browser_ui/4_verify/ci_job_artifacts/unlocking_job_artifacts_across_parent_child_pipelines_spec.rb:42 2026-01-08 (4 weeks) View Issue
qa/specs/features/browser_ui/4_verify/ci_job_artifacts/unlocking_job_artifacts_across_parent_child_pipelines_spec.rb:92 2026-01-08 (4 weeks) View Issue
qa/specs/features/browser_ui/4_verify/pipeline/merge_mr_when_pipline_is_blocked_spec.rb:59 2026-01-08 (4 weeks) View Issue
qa/specs/features/browser_ui/4_verify/ci_job_artifacts/unlocking_job_artifacts_across_parent_child_pipelines_spec.rb:67 2026-01-08 (4 weeks) View Issue
qa/specs/features/browser_ui/4_verify/ci_project_artifacts/user_can_bulk_delete_artifacts_spec.rb:33 2026-01-08 (4 weeks) View Issue
qa/specs/features/browser_ui/4_verify/pipeline/trigger_child_pipeline_with_manual_spec.rb:20 2026-01-08 (4 weeks) View Issue
spec/features/projects/pipelines/pipelines_spec.rb:165 2026-01-19 (3 weeks) View Issue
ee/spec/features/projects/pipelines/pipeline_spec.rb:256 2026-01-19 (3 weeks) View Issue
spec/features/projects/pipelines/pipelines_spec.rb:121 2026-01-19 (3 weeks) View Issue
ee/spec/features/merge_trains/user_adds_merge_request_to_merge_train_spec.rb:82 2026-01-19 (3 weeks) View Issue
spec/services/ci/refs/enqueue_pipelines_to_unlock_service_spec.rb:100 2026-01-19 (3 weeks) View Issue
spec/models/ci/pipeline_spec.rb:1506 2026-01-19 (3 weeks) View Issue
spec/serializers/test_report_summary_entity_spec.rb:23 2026-01-19 (3 weeks) View Issue
spec/features/projects/pipelines/pipelines_spec.rb:115 2026-01-19 (3 weeks) View Issue
spec/features/projects/pipelines/pipeline_spec.rb:845 2026-01-19 (3 weeks) View Issue
spec/services/ci/pipeline_triggers/create_service_spec.rb:81 2026-01-19 (3 weeks) View Issue
ee/spec/lib/ee/gitlab/cleanup/orphan_job_artifact_files_batch_spec.rb:30 2026-01-19 (3 weeks) View Issue
spec/features/merge_request/user_sees_pipelines_from_forked_project_spec.rb:57 2026-01-19 (3 weeks) View Issue
spec/services/object_storage/delete_stale_direct_uploads_service_spec.rb:71 2026-01-19 (3 weeks) View Issue
spec/services/object_storage/delete_stale_direct_uploads_service_spec.rb:94 2026-01-19 (3 weeks) View Issue
ee/spec/features/merge_trains/user_adds_merge_request_to_merge_train_spec.rb:125 2026-01-19 (3 weeks) View Issue
ee/spec/lib/ee/gitlab/cleanup/orphan_job_artifact_files_batch_spec.rb:38 2026-01-19 (3 weeks) View Issue
ee/spec/features/merge_trains/user_adds_merge_request_to_merge_train_spec.rb:63 2026-01-19 (3 weeks) View Issue
spec/serializers/test_report_summary_entity_spec.rb:17 2026-01-19 (3 weeks) View Issue
ee/spec/services/auto_merge/merge_train_service_spec.rb:326 2026-01-19 (3 weeks) View Issue
spec/services/ci/refs/enqueue_pipelines_to_unlock_service_spec.rb:109 2026-01-19 (3 weeks) View Issue
ee/spec/features/merge_trains/user_adds_merge_request_to_merge_train_spec.rb:195 2026-01-19 (3 weeks) View Issue
spec/services/object_storage/delete_stale_direct_uploads_service_spec.rb:47 2026-01-19 (3 weeks) View Issue
spec/models/concerns/ci/has_variable_spec.rb:64 2026-01-19 (3 weeks) View Issue
ee/spec/features/merge_trains/user_adds_merge_request_to_merge_train_spec.rb:183 2026-01-19 (3 weeks) View Issue
spec/models/ci/build_trace_chunks/fog_spec.rb:210 2026-01-19 (3 weeks) View Issue
spec/features/projects/pipelines/pipelines_spec.rb:131 2026-01-19 (3 weeks) View Issue
ee/spec/features/merge_trains/user_adds_merge_request_to_merge_train_spec.rb:168 2026-01-19 (3 weeks) View Issue
spec/workers/object_storage/delete_stale_direct_uploads_worker_spec.rb:19 2026-01-19 (3 weeks) View Issue
spec/requests/api/commit_statuses_spec.rb:125 2026-01-19 (3 weeks) View Issue
ee/spec/features/merge_trains/user_adds_merge_request_to_merge_train_spec.rb:101 2026-01-19 (3 weeks) View Issue
ee/spec/features/merge_trains/user_adds_merge_request_to_merge_train_spec.rb:216 2026-01-19 (3 weeks) View Issue
spec/features/projects/pipeline_schedules_spec.rb:138 2026-01-19 (3 weeks) View Issue
ee/spec/features/projects/pipelines/pipeline_spec.rb:125 2026-01-19 (3 weeks) View Issue
spec/lib/object_storage/pending_direct_upload_spec.rb:172 2026-01-19 (3 weeks) View Issue
spec/features/projects/pipelines/pipelines_spec.rb:177 2026-01-19 (3 weeks) View Issue
spec/requests/api/ci/pipelines_spec.rb:714 2026-01-19 (3 weeks) View Issue
spec/models/ci/pipeline_spec.rb:717 2026-01-19 (3 weeks) View Issue
ee/spec/features/merge_trains/user_adds_merge_request_to_merge_train_spec.rb:153 2026-01-19 (3 weeks) View Issue
qa/specs/features/ee/browser_ui/4_verify/root_cause_analysis_with_duo_chat_spec.rb:54 2026-01-27 (1 week) View Issue
spec/models/ci/pipeline_spec.rb:721 2026-02-02 (4 days) View Issue
spec/models/ci/pipeline_spec.rb:1510 2026-02-02 (4 days) View Issue
spec/features/projects/pipelines/pipelines_spec.rb:110 2026-02-04 (2 days) View Issue
spec/features/projects/pipelines/pipelines_spec.rb:156 2026-02-04 (2 days) View Issue
spec/features/projects/pipelines/pipelines_spec.rb:144 2026-02-04 (2 days) View Issue
spec/features/projects/pipelines/pipelines_spec.rb:94 2026-02-04 (2 days) View Issue
spec/features/projects/pipelines/pipeline_spec.rb:846 2026-02-04 (2 days) View Issue
spec/features/projects/pipelines/pipelines_spec.rb:100 2026-02-04 (2 days) View Issue

Related

Edited by David Dieulivol

Merge request reports