Improve flakyness and speed of pipeline
Overview
Merging to GitLab Runner is currently a pain for maintainers because pipelines are slow and flaky. Maintainers have to babysit each merge request to make sure it's merged.
Speed
Each pipeline ends up taking around 50minutes to run which is fairly slow of a feedback loop when trying to merge a lot of things together or trying to get a quick fix in. This is especially painful with merge trains since we first have to wait 1 hour to see if it passes the pipeline then add it to the merge train, which will run another pipeline before merging which is another 1 hour. Best case scenario, a merge request take about 2 hours to get merged which is VERY slow. The maintainer has to baby sit the pipeline to merge sure a merge request gets merged and we don't end up missing a merge request that we approved and never gets merged.
Flaky jobs
For example in !2085 (merged) it took us a few tries (and hours) to actually get it merged because of the flakiness:
Flaky pipelines are especially painful with merge trains because if a merge request fails inside of the merge train there is no way to add to the merge train before you run the pipeline again.
We track the flaky pipelines with failureflaky-test label