Change thresholds to apply the flakiness::* labels
With the help of my small script, I looked into the percentiles of flakiness reports for the latest 1000 updated failureflaky-test issues:
$ GITLAB_API_TOKEN="$(cat ~/gitlab/token-rymai.com)" be scripts/unhealthy_test_issues_statistics.rb --max-issues 1000 --health-problem-type flakiness
....................................................................................................Fetched 100 issues
....................................................................................................Fetched 200 issues
....................................................................................................Fetched 300 issues
....................................................................................................Fetched 400 issues
....................................................................................................Fetched 500 issues
....................................................................................................Fetched 600 issues
....................................................................................................Fetched 700 issues
....................................................................................................Fetched 800 issues
....................................................................................................Fetched 900 issues
....................................................................................................Fetched 1000 issues
P75: 3.0
P90: 21.0
P95: 41.0
{:number=>1000.0, :sum=>17299, :variance=>13330.58959899981, :standard_deviation=>115.45817250848816, :min=>1, :max=>2227, :mean=>17.299, :mode=>1, :median=>1.0, :range=>2226.0, :q1=>1.0, :q2=>1.0, :q3=>3.0}
Proposal
P75: 3.0 => less than or equal to 3 (current: less than or equal 9) => flakiness::4
P90: 21.0 => between 4 and 21 (current: between 10 and 499) => flakiness::3
P95: 41.0 => between 22 and 41 (current: between 500 and 999) => flakiness::2; more than 999 => flakiness::1
# of flakiness reports | percentile | Label |
---|---|---|
1-3 | < P75 | flakiness4 |
4-21 | P75 < x < P90 | flakiness3 |
22-41 | P90 < x < P95 | flakiness2 |
41+ | > P95 | flakiness1 |
Edited by Rémy Coutable