Skip to content

Speed up Elasticsearch AddNewDataToMergeRequestsDocuments migration

What does this MR do?

This migration was just introduced in !59836 (merged) .

After watching this migration running in production we can see that we can make it run faster. The numbers here also match what we used in AddPermissionsDataToNotesDocuments so should be safe.

I've also changed it to use the ProcessInitialBookkeepingService which is basically just moving the work into a different queue. We have 2 queues for indexing work in Elasticsearch and the "initial" queue is for newly created projects/imported projects. It gets much less traffic than the so called "incremental" queue and therefore it's safer to put backfill migrations into this queue. We actually should have used this queue to begin with but switching it now is perfectly fine.

At the currrent rate of 5000/5 minutes we have 15M remaining so it will take ~10 days.

With the updated rate of 6000/3 minutes it should take ~5 days.

Screenshots (strongly suggested)

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • Label as security and @ mention @gitlab-com/gl-security/appsec
  • The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • Security reports checked/validated by a reviewer from the AppSec team

Related to #322788 (closed)

Edited by Dylan Griffith

Merge request reports