Maintain Review App deployments success rate at 90%
In gitlab-org/gitlab-insights!81 (merged):
- I've written a script to analyze the success rate of any job.
- I've then added analysis for the
review-deploy
andreview-qa-smoke
jobs, from the start of the day before today in.gitlab-ci.yml
- Then I added these jobs to the scheduled pipelines here: https://gitlab.com/gitlab-org/gitlab-insights/pipeline_schedules
Periscope charts:
- GitLab: https://app.periscopedata.com/app/gitlab/496118/Engineering-Productivity-Sandbox?widget=6690556&udv=0
- GitLab Foss: https://app.periscopedata.com/app/gitlab/496118/Engineering-Productivity-Sandbox?widget=6783179&udv=0
Manually gathered historical data
The results starting from 2019-02-13 00:00:00 are as follows:
Starting from | Overall review-deploy success rate |
Overall review-qa-smoke success rate |
Notes |
---|---|---|---|
2019-02-13 | 80.94098883572569 | 39.76190476190476 | |
2019-02-14 | 81.21212121212122 | 53.93939393939394 | |
2019-02-15 | 84.74842767295598 | 69.33962264150944 | |
2019-02-16 | 63.75 | 54.166666666666664 | |
2019-02-17 | 91.66666666666667 | 77.1604938271605 | |
2019-02-18 | 78.11704834605598 | 62.213740458015266 | |
2019-02-19 | 78.36021505376344 | 68.54838709677419 | |
2019-02-20 | 62.375621890547265 | 57.71144278606965 | |
2019-03-12 | 85.29411764705883 | 3.587443946188341 | Review Apps are broken since 2019-03-06 due to a .gitlab-ci.yml change: https://gitlab.com/gitlab-org/gitlab-ce/issues/59113. |
2019-03-13 | 81.68168168168168 | 2.608695652173913 | Review Apps are broken since 2019-03-06 due to a .gitlab-ci.yml change: https://gitlab.com/gitlab-org/gitlab-ce/issues/59113. |
2019-03-14 | 85.41666666666667 | 2.459016393442623 | Review Apps are broken since 2019-03-06 due to a .gitlab-ci.yml change: https://gitlab.com/gitlab-org/gitlab-ce/issues/59113. |
2019-03-15 | 88.63636363636364 | 1.1363636363636365 | Review Apps are broken since 2019-03-06 due to a .gitlab-ci.yml change: https://gitlab.com/gitlab-org/gitlab-ce/issues/59113. |
2019-03-16 | 70.37037037037038 | 0.0 | Saturday, only 9 jobs. Review Apps are broken since 2019-03-06 due to a .gitlab-ci.yml change: https://gitlab.com/gitlab-org/gitlab-ce/issues/59113. |
2019-03-17 | 89.47368421052632 | 0.0 | Sunday. Review Apps are broken since 2019-03-06 due to a .gitlab-ci.yml change: https://gitlab.com/gitlab-org/gitlab-ce/issues/59113. |
2019-03-18 | 91.28440366972477 | 1.8181818181818181 | Review Apps are broken since 2019-03-06 due to a .gitlab-ci.yml change: https://gitlab.com/gitlab-org/gitlab-ce/issues/59113. |
2019-03-19 | 84.88372093023256 | 13.178294573643411 | This was fixed around 2019-03-19 2:19 UTC, and we can see that the success rate is already going up from 0-1% to 13%! |
2019-03-20 | 83.6021505376344 | 57.142857142857146 | Success rate is between 80% and 90% for approx. 9 days (except for the 16th which only had 9 jobs). QA success is going up! |
2019-03-21 | 80.47034764826175 | 59.50920245398773 | |
2019-03-22 | 68.23361823361823 | 51.70940170940171 | |
2019-03-23 | 70.83333333333333 | 57.69230769230769 | Saturday, 12 jobs. |
2019-03-24 | 100.0 | 90.47619047619048 | Sunday, 22 jobs. |
2019-03-25 | 79.1970802919708 | 63.13868613138686 | 137 jobs. |
2019-03-26 | 68.96207584830339 | 55.42168674698795 | 167 jobs. It seems, deployments are failing in a weird way. For instance, I see DNS records not being created (e.g. https://gitlab.com/gitlab-org/gitlab-ce/-/jobs/184989810). |
2019-03-27 | 39.03508771929825 | 31.543624161073826 | 152 jobs. It seems, deployments are failing in a weird way. For instance, I see DNS records not being created (e.g. https://gitlab.com/gitlab-org/gitlab-ce/-/jobs/184989810). |
2019-03-28 | 57.629107981220656 | 26.408450704225352 | 142 jobs. |
2019-03-29 | 87.62376237623762 | 56.60377358490566 | 101 jobs. |
2019-03-30 | 81.48148148148148 | 70.37037037037037 | Saturday, 27 jobs. |
2019-03-31 | 83.33333333333333 | 70.0 | Sunday, 42 jobs. |
2019-04-01 | 77.59740259740259 | 65.0326797385621 | 154 jobs. |
2019-04-02 | 78.34394904458598 | 68.78980891719745 | 157 pipelines, 287 jobs. Runners were overwhelmed and a lot of pipelines are pending, leading to upstream jobs (i.e. review-build-cng ) to fail, leading to review-deploy jobs to fail as well... |
2019-04-03 | 42.38095238095238 | 43.47826086956522 | 210 pipelines, 420 jobs. Runners were overwhelmed and a lot of pipelines are pending, leading to upstream jobs (i.e. review-build-cng ) to fail, leading to review-deploy jobs to fail as well... |
2019-04-04 | 68.26086956521739 | 29.07488986784141 | 230 pipelines, 317 jobs. A legit failure was introduced on 2019-04-03 18:21 UTC: gitlab-org/quality/staging#50 (closed). |
2019-04-05 | 73.02631578947368 | 36.18421052631579 | 152 pipelines, 204 jobs. A legit failure was introduced on 2019-04-03 18:21 UTC: gitlab-org/quality/staging#50 (closed). This was quarantined on 2019-04-05 7:52 UTC. A fix to make deployments for reliable was merged on 2019-04-05 8:42 UTC. |
2019-04-06 | 93.33333333333333 | 66.66666666666667 | 15 pipelines, 16 jobs. |
2019-04-07 | 88.88888888888889 | 83.33333333333333 | 18 pipelines, 20 jobs. |
2019-04-08 | 95.9% (raw: 82.78688524590164) | 66.66666666666667 | 122 pipelines, 147 jobs. |
2019-04-09 | 75.55555555555556 | 61.48148148148148 | 135 pipelines, 157 jobs. |
2019-04-10 | 26.380368098159508 | 20.245398773006134 | 163 pipelines, 204 jobs. It seems that we had a lot of parse error: Invalid numeric literal at line 1, column 10 in the review-build-cng job... |
2019-04-11 | 41.55844155844156 | 31.41025641025641 | 154 pipelines, 180 jobs. |
2019-04-12 | 87.70491803278688 | 80.83333333333333 | 122 pipelines, 126 jobs. |
2019-04-13 | 88.88888888888889 | 88.88888888888889 | 9 pipelines, 9 jobs. |
2019-04-14 | 81.25 | 75.0 | 16 pipelines, 16 jobs. |
2019-04-15 | 71.31782945736434 | 68.21705426356588 | 129 pipelines, 130 jobs. |
2019-04-16 | 89.6% (raw: 81.93548387096774) | 72.25806451612904 | 155 pipelines, 155 jobs. |
2019-04-17 | 73.55371900826447 | 60.65573770491803 | 121 pipelines, 130 jobs. |
2019-04-18 | 74.19354838709677 | 62.5 | 124 pipelines, 127 jobs. |
2019-04-19 | 86.3157894736842 | 78.57142857142857 | 95 pipelines, 108 jobs. |
2019-04-20 | 80.0 | 85.0 | 20 pipelines, 26 jobs. |
2019-04-21 | 77.77777777777777 | 55.55555555555556 | 9 pipelines, 11 jobs. |
2019-04-22 | 83.82352941176471 | 86.76470588235294 | 68 pipelines, 84 jobs. |
2019-04-23 | 77.4% (raw: 69.6078431372549) | 54.90196078431372 | 102 pipelines, 159 jobs. |
2019-04-24 | 88.5% (raw: 72.95081967213115) | 67.47967479674797 | 122 pipelines, 177 jobs. |
2019-04-25 | 56.79012345679013 | 54.43037974683544 | 81 pipelines, 147 jobs. |
2019-04-26 | 14.285714285714286 | 15.294117647058824 | 84 pipelines, 212 jobs. I found a probable reason why deployment would fail: https://gitlab.com/gitlab-org/quality/team-tasks/issues/93#note_164391286. Also, I were in the process of upgrading Kubernetes on both clusters. That should improve things. |
2019-04-27 | 8.333333333333334 | 16.666666666666668 | 12 pipelines, 31 jobs. I found a probable reason why deployment would fail: https://gitlab.com/gitlab-org/quality/team-tasks/issues/93#note_164391286. Also, I were in the process of upgrading Kubernetes on both clusters. That should improve things. |
2019-04-28 | 100.0 | 100.0 | 6 pipelines, 9 jobs. |
2019-04-29 | 98% (raw: 76.92307692307692) | 70.29702970297029 | 104 pipelines, 142 jobs. |
2019-04-30 | 74.15730337078652 | 57.30337078651685 | 89 pipelines, 131 jobs. |
2019-05-01 | 58.0 | 50.98039215686274 | 50 pipelines, 80 jobs. |
2019-05-02 | 82.53968253968254 | 70.3125 | 63 pipelines, 95 jobs. |
2019-05-03 | 83.01886792452831 | 71.69811320754717 | 53 pipelines, 75 jobs. |
2019-05-04 | 100.0 | 75.0 | 4 pipelines, 5 jobs. |
2019-05-05 | 100.0 | 100.0 | 3 pipelines, 3 jobs. |
2019-05-06 | 75.40983606557377 | 73.77049180327869 | 61 pipelines, 91 jobs. |
2019-05-08 | 93.5% (raw: 77.41935483870968) | 77.41935483870968 | 31 pipelines, 46 jobs. |
2019-05-09 | 73.33333333333333 | 71.42857142857143 | 15 pipelines, 21 jobs. |
2019-05-10 | 18.75 | 15.151515151515152 | 32 pipelines, 81 jobs. |
2019-05-11 | 25.0 | 25.0 | 12 pipelines, 28 jobs. |
2019-05-12 | 66.66666666666667 | 66.66666666666667 | 6 pipelines, 9 jobs. |
2019-05-13 | 97% (raw: 85.29411764705883) | 82.3529411764706 | 34 pipelines, 45 jobs. A single actual Error: forwarding ports: error upgrading connection: error dialing backend: ssh: rejected: connect failed (Connection timed out) Failed to init Tiller. failure. |
2019-05-14 | 97.5 | 97.43589743589743 | 40 pipelines, 44 jobs. |
2019-05-15 | 97.36842105263158 | 97.29729729729729 | 38 pipelines, 44 jobs. |
2019-06-12 | 69.86301369863014 | 65.27777777777777 | 73 pipelines, 74 jobs. |
2019-07-25 | 51.724137931034484 | 62.7906976744186 | 87 pipelines, 137 jobs. |
2019-07-26 | 52.17391304347826 | 52.22222222222222 | 92 pipelines, 142 jobs. |
2019-07-27 | 76.92307692307692 | 92.85714285714286 | 13 pipelines, 17 jobs. |
2019-07-28 | 100.0 | 85.71428571428571 | 7 pipelines, 7 jobs. |
2019-07-29 | 31.818181818181817 | 29.41176470588235 | 66 pipelines, 114 jobs. |
2019-07-30 | 71.875 | 71.42857142857143 | 64 pipelines, 87 jobs. https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/31238 was merged on that day. |
2019-07-31 | 62.6865671641791 | 53.73134328358209 | 67 pipelines, 106 jobs. |
2019-08-01 | 65.21739130434783 | 61.79775280898876 | 92 pipelines, 135 jobs. |
2019-08-02 | 69.49152542372882 | 40.67796610169491 | 59 pipelines, 85 jobs. |
2019-08-03 | 60.0 | 40.0 | 10 pipelines, 13 jobs. |
2019-08-04 | 85.71428571428571 | 0.0 | 7 pipelines, 7 jobs. |
2019-08-05 | 94.0 | 64.64646464646465 | 100 pipelines, 113 jobs. |
2019-08-06 | 80.85106382978724 | 72.04301075268818 | 94 pipelines, 128 jobs. |
2019-08-07 | 71.42857142857143 | 74.07407407407408 | 84 pipelines, 126 jobs. |
2019-08-08 | 85.48387096774194 | 75.86206896551724 | 62 pipelines, 79 jobs. |
2019-08-09 | 95.65217391304348 | 80.3030303030303 | 69 pipelines, 75 jobs. |
2019-08-10 | 100.0 | 83.33333333333333 | 11 pipelines, 11 jobs. |
2019-08-11 | 90.9090909090909 | 81.81818181818181 | 11 pipelines, 12 jobs. |
2019-08-12 | 94.25287356321839 | 74.71264367816092 | 87 pipelines, 103 jobs. |
2019-08-13 | 90.78947368421052 | 73.07692307692308 | 78 pipelines, 126 jobs. |
2019-08-14 | 85.24590163934427 | 82.75862068965517 | 61 pipelines, 82 jobs. |
2019-08-15 | 85.52631578947368 | 66.23376623376623 | 76 pipelines, 99 jobs. |
2019-08-16 | 86.25 | 85.8974358974359 | 80 pipelines, 104 jobs. |
2019-08-17 | 100.0 | 71.42857142857143 | 7 pipelines, 7 jobs. |
2019-08-18 | 100.0 | 85.71428571428571 | 7 pipelines, 7 jobs. |
2019-08-19 | 94.36619718309859 | 78.37837837837837 | 71 pipelines, 82 jobs. |
2019-08-20 | 76.92307692307692 | 70.0 | 91 pipelines, 117 jobs. |
2019-08-21 | 77.21518987341773 | 68.0 | 79 pipelines, 110 jobs. |
2019-08-22 | 74.48979591836735 | 67.67676767676768 | 98 pipelines, 131 jobs. |
2019-08-23 | 88.75 | 86.25 | 80 pipelines, 99 jobs. |
2019-08-24 | 88.88888888888889 | 88.88888888888889 | 9 pipelines, 12 jobs. |
2019-08-25 | 83.33333333333333 | 83.33333333333333 | 6 pipelines, 6 jobs. |
2019-08-26 | 98.86363636363636 | 91.86046511627907 | 88 pipelines, 93 jobs. |
2019-08-27 | 80.0 | 74.07407407407408 | 80 pipelines, 107 jobs. |
2019-08-28 | 65.82278481012658 | 57.69230769230769 | 79 pipelines, 116 jobs. |
2019-08-29 | 92.13483146067416 | 69.66292134831461 | 89 pipelines, 106 jobs. |
2019-08-30 | 76.84210526315789 | 59.79381443298969 | 95 pipelines, 133 jobs. |
2019-08-31 | 66.66666666666667 | 44.44444444444444 | 6 pipelines, 8 jobs. |
2019-09-01 | 88.88888888888889 | 100.0 | 9 pipelines, 9 jobs. |
2019-09-02 | 96.3855421686747 | 76.3157894736842 | 83 pipelines, 90 jobs. |
2019-09-03 | 57.142857142857146 | 56.09756097560975 | 84 pipelines, 119 jobs. |
2019-09-04 | 53.535353535353536 | 61.224489795918366 | 99 pipelines, 176 jobs. |
2019-09-05 | 0.9345794392523364 | 5.319148936170213 | 107 pipelines, 203 jobs. |
2019-09-06 | 4.724409448818897 | 4.705882352941177 | 127 pipelines, 235 jobs. |
2019-09-07 | 20.0 | 60.0 | 10 pipelines, 21 jobs. |
2019-09-08 | 13.333333333333334 | 25.0 | 15 pipelines, 26 jobs. |
2019-09-09 | 27.659574468085108 | 34.146341463414636 | 141 pipelines, 255 jobs. |
2019-09-10 | 5.202312138728324 | 13.513513513513514 | 173 pipelines, 328 jobs. |
2019-09-11 | 5.594405594405594 | 6.315789473684211 | 143 pipelines, 215 jobs. |
2019-09-12 | 17.15686274509804 | 14.864864864864865 | Switched to Single Repo 74 pipelines, 89 jobs. |
2019-09-13 | 67.77777777777777 | 63.63636363636363 | 90 pipelines, 92 jobs. |
2019-09-14 | 0.0 | 0.0 | 20 pipelines, 20 jobs. |
2019-09-15 | 0.0 | 0.0 | 30 pipelines, 30 jobs. |
2019-09-16 | 12.682926829268293 | 7.5 | 205 pipelines, 208 jobs. |
2019-09-17 | 23.404255319148938 | 30.8411214953271 | 235 pipelines, 244 jobs. |
2019-09-19 | 44.84304932735426 | 45.794392523364486 | 223 pipelines, 230 jobs. |
2019-09-20 | 33.55263157894737 | 46.96969696969697 | 152 pipelines, 152 jobs. |
2019-09-21 | 91.42857142857143 | 90.9090909090909 | 35 pipelines, 35 jobs. |
2019-09-22 | 100.0 | 70.0 | 16 pipelines, 17 jobs. |
Edited by Rémy Coutable