Verify EP pipeline dashboards - 2024-04-26
Goal
Verify that the Engineering Productivity dashboards display accurate data.
The start of this process can be found on this RUNBOOK entry.
Resources to check
-
SQL queries behind the Pipeline duration dashboard -
Pipeline duration dashboard -
Tableau pipeline duration dashboard
Steps
-
Review the SQL queries for each widget in the dashboards. -
For pipeline dashboards with types, check that the CI/CD job names we use to define the types are still used in the CI/CD config. -
If they aren't used anymore, check when was the last time they were used (use git blame
and the like to find out), and:-
If the CI/CD job name wasn't used for more than 1 year, remove them, and create MRs to change the following: -
SQL queries in https://gitlab.com/gitlab-org/quality/engineering-productivity/snowflake-dashboard-sql -
The scripts/pipelines-select
script in theteam
project. -
The scripts/pipeline/set-pipeline-name
script ingitlab-org/gitlab
.
-
-
If the CI/CD job name was used less than a year ago, add a comment in the scripts to remove the CI/CD job references 1 year after it was removed from the CI/CD config (example).
-
-
-
Have all the MRs merged, and update the SQL queries in Snowflake, if applicable. -
Compare the Snowflake dashboards and the output of the scripts/pipelines-select
scripts. They should be identical. -
Compare the Snowflake dashboards with the Tableau dashboards. They should be identical. -
Add your results under the Results
section below.
Here is an example of a past verification.
Results
SQL queries behind the Pipeline duration dashboard
There were some deprecated CI/CD jobs we used in the queries, so I removed them and updated the other locations:
- !206 (merged)
- https://gitlab.com/gitlab-org/quality/engineering-productivity/snowflake-dashboard-sql/-/merge_requests/47
- gitlab-org/gitlab!151666 (merged)
Pipeline duration dashboard
Daily
Ruby script
$ scripts/pipelines-select --after "2024-04-17 00:00:01" --before "2024-04-17 23:59:59"
[...]
###################################
############# Summary #############
###################################
Number of pipelines: 416
Average duration: 70.46
P50 duration: 42.43
P80 duration: 142.3
P90 duration: 152.29
Weekly
Ruby script
$ scripts/pipelines-select --after "2024-04-15 00:00:01" --before "2024-04-21 23:59:59"
[...]
###################################
############# Summary #############
###################################
Number of pipelines: 2070
Average duration: 69.16
P50 duration: 47.65
P80 duration: 134.9
P90 duration: 149.1
Monthly
Ruby script
$ scripts/pipelines-select --after "2024-03-01 00:00:01" --before "2024-03-31 23:59:59"
[...]
###################################
############# Summary #############
###################################
Number of pipelines: 5621
Average duration: 61.2
P50 duration: 52.2
P80 duration: 103.2
P90 duration: 122.4
Pipelines with types dashboards
I picked 2024-03-01
to check the pipeline with types widgets.
I also copied the pipeline duration with types widgets, and ran them for the day I chose:
-- Before
AND pipelines.finished_at >= DATEADD('month', -1, CURRENT_DATE)
-- After
AND finished_day = 'YYYY-MM-DD'
For the histogram, I picked three buckets at random: 6, 54 and 97.
Ruby script
$ scripts/pipelines-select --after "2024-03-01 00:00:01" --before "2024-03-01 23:59:59" --with-types
[...]
###################################
############# Summary #############
###################################
[...]
Pipeline count by types:
docs: 96
code: 13
rspec_predictive: 119
e2e_gdk: 24
review_app: 3
e2e: 144
: 1
Pipeline Histogram by duration:
[...]
6:
docs: 76
[...]
54:
rspec_predictive: 3
code: 2
e2e_gdk: 1
[...]
97:
e2e: 9
e2e_gdk: 1
Tableau pipeline duration dashboard
Edited by David Dieulivol