## Context * See [this approach](https://gitlab.com/gitlab-com/gitlab-OKRs/-/work_items/5806#our-approach) to reduce MR pipeline duration. * [61% of the pipelines](https://app.snowflake.com/ys68254/gitlab/w2INmdSZ6iw2/chart) we ran in February 2024 were E2E (38%) and E2E-GDK (23%) pipelines. ## Goals * Try to **run time-consuming pipelines less often**. ## [Feedback issue](https://gitlab.com/gitlab-org/quality/engineering-productivity/team/-/issues/500) ## Idea to discuss Initially brought up by @godfat-gitlab in [this comment](https://gitlab.com/gitlab-org/quality/engineering-productivity/team/-/issues/428#note_1783576278): > Yes I agree. I think we may think about tiered pipelines, that we incrementally run more as the merge request progresses. We could have **pipeline tiers** that would run more and more long-running jobs are the MR progresses in its lifecycle (e.g. first approval given, all approvals are given, MR is set to auto-merge, ...). **Tier:** A **group of pipeline types** (e.g. RSpec Predictive + Jest + E2E-GDK) ### Sample Proposal * **Tier 1**: MR has no approvals - Run lightweight predictive pipelines * **Tier 2**: MR has at least one approval, but not all of them - Run predictive pipelines, maybe E2E-GDK tests * **Tier 3**: MR has all the approvals it needs - Run full pipelines and E2E-GDK tests ### Thoughts for implementation * We could implement all of this logic using **labels** like `~"mr-pipeline:tier-1"`, `~"mr-pipeline:tier-2"`, which would be a generalization of team~28119070, and they could be managed via triage-ops. * **Going to lower tiers as soon as possible** is a key element to **run less time-consuming pipelines** (e.g. when approvals are removed, go back to tier-1) * The way we structure tiers will be very important, and probably tricky at times. We should be careful about the frustration of end-users (e.g. errors happen too often in the last tier - see next point) * This concept would allow us to have a more granular **tier-based monitoring**: * **At which tiers are errors happening the most?** (we should strive to have most of the errors as early as possible in the tiers) * **In which tiers are MRs in the most?** * **Duration of pipelines in a certain tier?** ## :nerd: More data on MRs and their pipeline types See [the number of pipeline types we run per MRs](https://app.snowflake.com/ys68254/gitlab/w5RYukhYKP4L):  [This MR](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/145685) is particularly interesting to look at :boom: Analysing this data a bit further using percentiles with the script below, we get the following data: <details><summary>percentiles script</summary> ```shell require 'descriptive_statistics' require 'csv' filename = '/Users/gitlab/Downloads/[Last 3 months] Merged MRs with the kind of pipelines they were running.csv' ['total', 'qa', 'frontend', 'qa-gdk', 'rspec-predictive', 'code', 'docs'].each do |column_name| csv = CSV.read(filename, headers: true) puts "#{column_name} pipelines:" [99, 95, 90, 80, 50].each do |percentile| puts "\t P#{percentile}: #{csv[column_name].percentile(percentile).round(2)}" end end ``` </details> ```shell $ ruby stats-script.rb total pipelines: P99: 21.0 P95: 12.0 P90: 9.0 P80: 6.0 P50: 3.0 qa pipelines: P99: 18.0 P95: 10.0 P90: 7.0 P80: 5.0 P50: 2.0 frontend pipelines: P99: 0.0 P95: 0.0 P90: 0.0 P80: 0.0 P50: 0.0 qa-gdk pipelines: P99: 0.0 P95: 0.0 P90: 0.0 P80: 0.0 P50: 0.0 rspec-predictive pipelines: P99: 8.0 P95: 3.15 P90: 2.0 P80: 1.0 P50: 0.0 code pipelines: P99: 3.0 P95: 1.0 P90: 0.0 P80: 0.0 P50: 0.0 docs pipelines: P99: 10.0 P95: 4.0 P90: 2.0 P80: 1.0 P50: 0.0 ``` E2E pipelines are the ones we could optimize the most. Currently, the P50 is at **2**. I would expect **the P80 or the P90 to be at 2 instead**, and we should ideally not have the P99 at more than 2 or 3.