Follow up - Benchmark for 'Provide a way to easily start multiple manual '
After 28741 - Provide a way to easily start multiple manual jobs, we have two possible ways to start manual actions in a stage:
- By clicking "Play all manual" button per stage (introduced in https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/27188)
- By individually clicking each manual action in a stage.
1. 'Play all manual' button in a stage | 2. 'Play' button in manual action |
---|---|
We need to do a comparison between the two to examine how much time can be saved by clicking one button per stage (Play all manual) vs clicking each manual action individually.
Proposal.
- With the same structure of our Omnibus Pipelines (i.e. https://dev.gitlab.org/gitlab/omnibus-gitlab/pipelines/1112403) as a template, write two capybara scripts:
- One that clicks on 'Play all manual' button in a stage
- Another one that clicks 'Play' in each manual action in a stage
- Benchmark both scripts
- Write down the difference and analyze.
Analysis
Scripts
- Script for clicking 'Play all manual' button per stage.
- Script for individually clicking each manual action in a stage.
- Script that runs a benchmark on both scripts.
Pipeline
- Script 1 and 2 used a simulated version of the Omnibus pipeline.
- This pipeline has two manual stages:
Staging_upload
with 6 manual actions andPackage_and_image
with 9 manual actions - 15 manual actions in total
Pipeline Screenshot |
---|
Benchmark
Time it took to click the 'Play all manual' button per stage (two buttons) vs clicking 15 manual actions:
Script | user | system | total | real |
---|---|---|---|---|
Clicking 'Play all manual' per stage | 0.022172 | 0.044576 | 0.066748 | 0.112180 |
Clicking 'Play' per manual action | 1.009409 | 0.604544 | 1.613953 | 1.367849 |
Results
It took a little over 100 milliseconds
to click two 'Play all manual' buttons vs 1.36 seconds
to click each individual build.
Applying that to the latest release candidates for GitLab.com:
Milestone | # RC | Time to start all manual builds with Play all manual (# RC * 0.112180 ) |
Time to start all manual actions individually (# RC * 1.367849 ) |
---|---|---|---|
11.11 | 6 Release Candidates (so far) | 0.67308 | 8.207094000000001 |
11.10 | 48 Release Candidates | 5.38464 | 65.65675200000001 |
11.9 | 46 Release Candidates | 5.16028 | 62.921054000000005 |
Based on the data, we can infer that activating all manual actions at once per stage is around 12
times faster, than activating individual manual actions. Hopefully, the addition of this button will save some valuable minutes from Release Managers