[VSA] Add worker for checking data consistency
The new value stream analytics tables are periodically aggregated. This means that data changes in the "core" tables (issues
, merge_requests
) might be missed and we would show incorrect data.
Goal: implement a worker that scans the root group, detects and fixes inconsistent records.
Records are inconsistent when:
- The aggregated tables (
analytics_cycle_analytics_issue_stage_events
,analytics_cycle_analytics_merge_request_stage_events
) contain data that is not present in the core tables.- Fix: delete these rows
- The aggregated tables contain wrong data (missing or outdated _event_timestamp).
- Fix: upsert these rows
We currently collect the data using the Analytics::CycleAnalytics::GroupDataLoaderWorker
class. We could extend this class to have a "mode" option which could optionally do the consistency check. The worker already knows how to iterate over the records, so we just need to implement the compare and cleanup logic.
Edited by Adam Hegyi