Skip to content

Updating worker to use keyset pagination

Max Fan requested to merge 432814-use-keyset-pagination-for-stuckImportWorker into master

What does this MR do and why?

Background: !138387 (comment 1677166637)

The current .find_each method does not work well with indexes that use timestamps. We'll need to change this to .each_batch to utilize the indexes properly

Performance Calculations

Before Indexes & MR Change

Import: https://console.postgres.ai/gitlab/gitlab-production-tunnel-pg12/sessions/24480/commands/78143

Entity: https://console.postgres.ai/gitlab/gitlab-production-tunnel-pg12/sessions/24480/commands/78144

After Indexes & MR Change

Import: https://console.postgres.ai/gitlab/gitlab-production-tunnel-pg12/sessions/24478/commands/78137

Entity: https://console.postgres.ai/gitlab/gitlab-production-tunnel-pg12/sessions/24478/commands/78138

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

Before After

How to set up and validate locally

  1. StuckImportWorker should behave as usual
  2. Create some imports, update the scope or the updated_at of the objects to 3.days.ago
  3. running StuckImportWorker should kill/timeout these outdated imports

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #432814 (closed)

Edited by Max Fan

Merge request reports