Experiment with ci_atomic_processing and ci_composite_status for GitLab project
Objective
Experiment with ci_atomic_processing
and ci_composite_status
in the gitlab-org/gitlab
pipelines.
Background
In an experiment with gitlab-foss
there was significant improvement in duration, DB, CPU for PipelineProcessWorker
.
- 85% faster average duration times for PipelineProcessWorker
- 105% faster average db_duration for PipelineProcessWorker
- 36% less cpu-s
Since 2020-01-01, gitlab-foss
has about 11 pipelines run per day and little to no concurrency and gitlab
has 484 per day. Source
Plan
Date & time | Activity | DRI |
---|---|---|
2020-04-24 12:00:00 UTC | Initial pilot: Enable feature flags for gitlab-org/gitlab
|
@ayufan |
2020-04-24 12:00:00 UTC | Initial pilot: Monitor pipeline success rate | @rymai |
2020-04-24 15:00:00 UTC | Initial pilot: Disable feature flags for gitlab-org/gitlab
|
@ayufan |
2020-04-26 | Initial pilot: Review resource utilization in Kibana Dashboards | @nmilojevic1 |
2020-04-27 09:00:00 UTC | Enable feature flags for gitlab-org/gitlab
|
@ayufan |
2020-04-27 09:00:00 UTC | Monitor pipeline success rate | @rymai and Engineering Productivity |
2020-04-30 14:00:00 UTC | Disable feature flags for gitlab-org/gitlab
|
@ayufan |
2020-05-04 | Report on resource utilization summary | @nmilojevic1 |
Monitoring
- #master-broken for master pipeline stability
- Pipeline Worker Process Dashboard
- Pipelines API Dashboard
Risks
-
Stability of pipelines decreases with larger pipeline load and concurrency than gitlab-org/gitlab-foss
Backout plan
- If pipeline success rate is being impacted then the feature flags will be disabled and the plan will be re-evaluated.
/chatops run feature set --project=gitlab-org/gitlab ci_atomic_processing false
/chatops run feature set --project=gitlab-org/gitlab ci_composite_status false
Success Criteria
- Decrease resource utilization for
PipelineProcessWorker
- Little to no decrease in master or merge request stability
- (bonus) Some decrease in pipeline duration
Edited by Kyle Wiebers