ConsistencyWorker exceeds 300 seconds

Problem

Analytics::CycleAnalytics::ConsistencyWorker is exceeding 300 seconds and takes the most out of our error budget.

Proposal

&7839 (comment 900630602)

In the consistency worker, we limit the maximum runtime to 5 minutes however, we do the check after one "item" (group) is finished processing. The processing of one item can be longer than 5 minutes.

  1. Iterate each aggregated issues and MRs
  2. If the issue or MR is already deleted, remove the aggregated row.

For large groups, this will take some time. We can fix it by tracking the execution. When the 5-minute limit is done, we stop the processing. The next job would continue the processing where the previous job finished.

Edited by Michelle Gill