Fix DORA deployment frequency calculation
What does this MR do and why?
The DORA deployment frequency calculation is incorrect for the following time intervals: all
, month
. The calculation is correct for day
aggregations because the query calculates SUM(frequency)
.
Problem: The current aggregation query returns the total number of deployments for the given time range.
The correct calculation is the following:
DEPLYOMENT_COUNT / NUMBER_OF_DAYS
The fix is behind a feature flag fix_dora_deployment_frequency_calculation
which will be enabled in 16.0 since it's a rather big change.
Implementation
To fix the metrics I decided to post-process the deployment counts and use the passed in start and end date to determine the time range. It's also possible to fix this on the SQL level, however it requires a much bigger refactor since date ranges are not passed to the SELECT
query. Additionally, bucketing (monthly) adds quite a bit of complexity in SQL.
The change affects the following features:
- DORA API
- VSA (deployment frequency was already calculated there correctly, now we're calculating the count from provided frequency)
- Note: the changed classes here will go away once we switch to the
flowMetrics
GraphQL query. The calculations inflowMetrics
have been already corrected.
- Note: the changed classes here will go away once we switch to the
- Insights (uses the DORA API)
How to test it:
- Ensure you're on ultimate plan.
- Seed VSA:
SEED_CYCLE_ANALYTICS=true SEED_VSA=true FILTER=cycle_analytics rake db:seed_fu
- Create DORA deployment:
FactoryBot.create(:dora_daily_metrics, date: 1.week.ago, deployment_frequency: 10, environment: FactoryBot.create(:environment, :production, project: Project.last))
- Go to the project-vsa page: Analytics -> Value Stream
- See if the deployment count and frequency metrics shows up
- Enable the
fix_dora_deployment_frequency_calculation
FF and verify that there is no change.
Where the change is visible:
- Do the steps in the previous list
- Go to Project -> Insights
- Select DORA from the dropdown
- Without the FF, you should see 10 for the DF metric (monthly chart)
- With the FF, you should see around 0.3 (<0.5) for the DF metric (monthly chart)
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Related to #394712 (closed)