Skip to content

Fix duplicate finished build sync events

Pedro Pombeiro requested to merge pedropombeiro/428146/remove-duplicates-from-batches into master

What does this MR do and why?

This MRs removes 3 feature flags that were used to debug the duplicates being added to the ci_finished_builds_aggregated_queueing_delay_percentiles_mv materialized view, and fixes the problem with sending duplicate build IDs to ClickHouse. We do this by loading the AR relationship before calling pluck(:build_id), so that the call to last doesn't generate an extra query.

EE: true

Part of #428146 (closed)

This change was tested in the production console, where it was confirmed that @processed_record_ids contained duplicate builds IDs at the end of execute.

image

The builds in @processed_build_ids looked like the following, if split by batch:

@processed_build_ids by batch 
[[
  # ... 498 unique IDs,
  5_634_258_445,
  5_634_259_020],
[
  5_634_258_445,
  5_634_259_020,
  # ... 494 unique IDs,
  5_634_474_423,
  5_634_474_548,
  5_634_474_559,
  5_634_474_885],
 [5_634_474_423,
  5_634_474_548,
  5_634_474_559,
  5_634_474_885,
  # ... 496 unique IDs
 ],
 [ # ... 496 unique IDs,
  5_634_583_521,
  5_634_583_562,
  5_634_583_577,
  5_634_583_579,
  5_634_583_582,
  5_634_583_584],
 [5_634_583_521,
  5_634_583_562,
  5_634_583_577,
  5_634_583_579,
  5_634_583_582,
  5_634_583_584,
  # ... 492 unique IDs,
  5_634_614_703,
  5_634_614_785],
 [5_634_614_703,
  5_634_614_785,
  # ... 495 unique IDs,
  5_634_636_196,
  5_634_636_225,
  5_634_636_274],
 [5_634_636_196,
  5_634_636_225,
  5_634_636_274,
  # ... 496 unique IDs,
  5_634_651_184],
 [5_634_651_184,
  # ... 492 unique IDs,
  5_634_667_919,
  5_634_667_925,
  5_634_667_929,
  5_634_667_934,
  5_634_667_943,
  5_634_668_045,
  5_634_668_092],
 [5_634_667_919,
  5_634_667_925,
  5_634_667_929,
  5_634_667_934,
  5_634_667_943,
  5_634_668_045,
  5_634_668_092,
  # ... 489 unique IDs,
  5_634_680_068,
  5_634_680_069,
  5_634_680_072,
  5_634_680_076],
 [5_634_680_068,
  5_634_680_069,
  5_634_680_072,
  5_634_680_076,
  # ... 491 unique IDs,
  5_634_688_597,
  5_634_688_600,
  5_634_688_603,
  5_634_688_607,
  5_634_688_608],
 [5_634_688_597,
  5_634_688_600,
  5_634_688_603,
  5_634_688_607,
  5_634_688_608,
  # ... 493 unique IDs,
  5_634_693_846,
  5_634_693_852],
 [5_634_693_846,
  5_634_693_852,
  # ... 349 unique IDs
 ],
 []]

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

n/a

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

n/a

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Pedro Pombeiro

Merge request reports