Skip to content

Fix DORA deployment frequency calculation

Adam Hegyi requested to merge 394712-fix-dora-df-metrics into master

What does this MR do and why?

The DORA deployment frequency calculation is incorrect for the following time intervals: all, month. The calculation is correct for day aggregations because the query calculates SUM(frequency).

Problem: The current aggregation query returns the total number of deployments for the given time range.

The correct calculation is the following:

DEPLYOMENT_COUNT / NUMBER_OF_DAYS

The fix is behind a feature flag fix_dora_deployment_frequency_calculation which will be enabled in 16.0 since it's a rather big change.

Implementation

To fix the metrics I decided to post-process the deployment counts and use the passed in start and end date to determine the time range. It's also possible to fix this on the SQL level, however it requires a much bigger refactor since date ranges are not passed to the SELECT query. Additionally, bucketing (monthly) adds quite a bit of complexity in SQL.

The change affects the following features:

  • DORA API
  • VSA (deployment frequency was already calculated there correctly, now we're calculating the count from provided frequency)
    • Note: the changed classes here will go away once we switch to the flowMetrics GraphQL query. The calculations in flowMetrics have been already corrected.
  • Insights (uses the DORA API)

How to test it:

  1. Ensure you're on ultimate plan.
  2. Seed VSA: SEED_CYCLE_ANALYTICS=true SEED_VSA=true FILTER=cycle_analytics rake db:seed_fu
  3. Create DORA deployment:
     FactoryBot.create(:dora_daily_metrics, date: 1.week.ago, deployment_frequency: 10, environment: FactoryBot.create(:environment, :production, project: Project.last))
  4. Go to the project-vsa page: Analytics -> Value Stream
  5. See if the deployment count and frequency metrics shows up
  6. Enable the fix_dora_deployment_frequency_calculation FF and verify that there is no change.

Where the change is visible:

  1. Do the steps in the previous list
  2. Go to Project -> Insights
  3. Select DORA from the dropdown
  4. Without the FF, you should see 10 for the DF metric (monthly chart)
  5. With the FF, you should see around 0.3 (<0.5) for the DF metric (monthly chart)

image

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #394712 (closed)

Edited by Adam Hegyi

Merge request reports

Loading