Feature evaluation: Merge Trains for www-gitlab-com
Summary
Enable Merge Trains on www-gitlab-com project in order to make sure the feature is scalable enough for semi-large repository.
If you have a question, please ask in f_merge_trains or s_release Slack channels.
Feature details
Please read https://docs.gitlab.com/ee/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.html for learning more about Merge Train.
Read more
Using Pipelines for Merged Results itself doesn't contribute to the CI cost increase, as the number of created pipelines won't be different. Although, this could slightly increase Gitaly load because it passesrecheck to MergeabilityCheckService, so we'd better to monitor the load.
Using Merge Train could increase CI cost, as it's basically creates additional pipeline per merging an MR, so basically we can estimate the increase diff with this simple formula - daily_increase = per-pipeline-cost * daily_merged_mr_count. There is an edge case that when a merge request has been dropped from a merge train, pipeline reconstruction happens on the train, however, this reconstraction happens maximum four pipelines per train, which means we wouldn't observe significant CI cost increase unless pipeline failures on merge train happen frequently.
Performance-wise, it aslo slightly increases Gitaly load because it generates train-ref per merge request on train. Also, we should look at the sidekiq worker AutoMergeProcessWorker's activity https://dashboards.gitlab.net/d/000000124/sidekiq-workers?orgId=1&refresh=5s&var-worker=AutoMergeProcessWorker%23perform&var-database=influxdb-01-inf-gprd closely as it's the main sidekiq job for processing merge train orchestration.
Lastly, we've already started working on dogfooding on CE/EE (See https://gitlab.com/gitlab-org/gitlab-ce/issues/57190, https://gitlab.com/gitlab-org/quality/team-tasks/issues/195 and https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/15761). I'd suggest we start dogfooding on www-gitlab-com for measuring/collecting some metrics and making sure that it's enough performant on CE/EE, as I'd expect CE/EE would introduce bigger impact than that.
If we observe troubles, problems, or unacceptable-performance/cost-increase, we can immediately stop using the above features by simply disabling the checkbox https://docs.gitlab.com/ee/ci/merge_request_pipelines/pipelines_for_merged_results/#enabling-pipelines-for-merged-results. All merge requests will be dropped from merge train immediately and go back to the previous behavior
Timeline
- 2019-12-?? 00:00 UTC: Will enable on www-gitlab-com for 1-4 hours gitlab-org/gitlab#43306 (closed)
- 2019-10-31 02:00 UTC: Disabled on www-gitlab-com (Second evaluation)
- 2019-10-23 07:00 UTC: Enabled on www-gitlab-com (Second evaluation)
- 2019-09-09 22:00 UTC: Disabled on www-gitlab-com (First evaluation)
- 2019-09-06 19:00 UTC: Enabled on www-gitlab-com (First evaluation)
How to enable the feature
- Visit your project’s Settings > General and expand Merge requests.
- Check Merge pipelines will try to validate the post-merge result prior to merging.
- Click Save changes button.
How to disable the feature
- Visit your project’s Settings > General and expand Merge requests.
- Check-Off Merge pipelines will try to validate the post-merge result prior to merging.
- Click Save changes button.
How to collect metrics
- Waiting time per merge request => https://gitlab.com/snippets/1923867
- Direct merge without merge train. => https://gitlab.com/snippets/1923868
Beginning of evaluation
-
Announce in Slack/CompanyCall (T-minus 1 day) -
Enable the feature
End of evaluation
-
Announce in Slack/CompanyCall -
Disable the feature
Feedback/Metrics
The second evaluation
We've figured out we have the following problem. We'll fix these problems before we move on to the next evaluation.
The first evaluation
We've figured out we have the following problem. We'll fix these problems before we move on to the next evaluation.