Use deduplicated finder in CI:JobAnalytics during backfill

Narendranrequested to merge
narendran/ci-job-analytics-api-patch into master

What does this MR do and why?

Use the deduplicated finder in CI:JobAnalytics during backfill

  • Use FinishedBuildsDeduplicatedFinder in Ci::JobAnalytics::QueryBuilder to ensure consistent query results when bbm is running
  • The implementation uses a Redis flag in Gitlab::Redis::SharedState that should be set when the BBM starts and cleared when it finalizes

next MR changes would look like this ⬇️ (backfill script):

In the BBM queue migration's up method:

ClickHouse::MigrationSupport::CiFinishedBuildsConsistencyHelper.set_backfill_in_progress!

In the finalize migration's up method:

ClickHouse::MigrationSupport::CiFinishedBuildsConsistencyHelper.clear_backfill_in_progress!

References

Related to #585201 (closed)

How to set up and validate locally 

# 1. Check initial state - should use regular finder
ClickHouse::MigrationSupport::CiFinishedBuildsConsistencyHelper.backfill_in_progress?
# => false

# 2. Verify QueryBuilder uses regular finder
project = Project.first
user = project.users.first
builder = Ci::JobAnalytics::QueryBuilder.new(project: project, current_user: user, options: { select_fields: [:name] })
builder.send(:base_finder).class.name
# => "ClickHouse::Finders::Ci::FinishedBuildsFinder"

# 3. Simulate BBM starting - set the Redis flag
ClickHouse::MigrationSupport::CiFinishedBuildsConsistencyHelper.set_backfill_in_progress!

# 4. Verify flag is set
ClickHouse::MigrationSupport::CiFinishedBuildsConsistencyHelper.backfill_in_progress?
# => true

# 5. Verify QueryBuilder now uses deduplicated finder
builder = Ci::JobAnalytics::QueryBuilder.new(project: project, current_user: user, options: { select_fields: [:name] })
builder.send(:base_finder).class.name
# => "ClickHouse::Finders::Ci::FinishedBuildsDeduplicatedFinder"

# 6. Simulate BBM finalizing - clear the Redis flag
ClickHouse::MigrationSupport::CiFinishedBuildsConsistencyHelper.clear_backfill_in_progress!

# 7. Verify flag is cleared and regular finder is used again
ClickHouse::MigrationSupport::CiFinishedBuildsConsistencyHelper.backfill_in_progress?
# => false
builder = Ci::JobAnalytics::QueryBuilder.new(project: project, current_user: user, options: { select_fields: [:name] })
builder.send(:base_finder).class.name
# => "ClickHouse::Finders::Ci::FinishedBuildsFinder"

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Merge request reports