Skip to content

Validate: Remove Long Running test-on-omnibus Pipeline Setup for Backport MRs

Problem

Validate if e2e:test-on-omnibus-ee is executed on backport MRs, and transition them if so.

As identified in gitlab-org/quality/quality-engineering/team-tasks#3623 (closed) -

End-to-end suites

  • e2e:test-on-omnibus-ee full suite in tier 3
  • qa:test-on-gdk gdk-instance suite in tier 3

Context

Omnibus and CNG pipelines have significantly different runtimes

From our pipeline efficiency track:

  • ⏲️ ~95 minutes saved per MR pipeline on its critical path (~140 mins omnibus -> ~45 mins cng Dashboard)
  • 💵 ~10'000USD/month saved (calculation)
  • 🧱 Less pipeline false failures due to ability to block MRs based on test failures. Previously this gate was loosely added.

Proposal

To save time, money, and false failures, transition any e2e:test-on-omnibus type pipelines to e2e:test-on-cng if possible.

Metrics

Baseline e2e:test-on-omnibus-ee as of 2024-05-02:

Tier 1 (ref): ~95m Tier 3 (ref): ~95m

Tasks

Results

Task Result
Removed e2e:test-on-omnibus-ee from tier-1 backport MR pipelines Reduces tier 1 pipeline runtime ~40% without compromising quality.
Removed instance test suite (main e2e test suite) from test-on-omnibus Reduces costs by not running same test suite against several deployment types, reduces flaky failure probability
Removed test-on-gdk and replaced it with test-on-cng Higher confidence in patch release as it now uses actual customer like deployment instead of dev deployment, P80 feedback time improvement by 20 minutes which is the difference between test-on-cng and test-on-gdk pipeline runtimes
Edited by andrey