Skip to content

Migration to move merge requests to separate Elasticsearch index

What does this MR do?

This is the last step in &5468 (closed) to move merge requests out of the main monolithic Elasticsearch index into a dedicated index. This will allow searches to be much faster.

This MR contains a new migration which copies all the merge requests from the main index into the new index. Then there is also code which checks if this new migration is completed and from there on sends all reads and writes to the new index.

Since it's tricky to write a zero downtime migration this migration will pause indexing while it is running.

The expected time taken can be estimated as:

  1. We have 18M merge requests in the index
  2. The migration breaks out the reindexing into 120 slices (this is how many shards we have in the main index)
  3. The slices are executed 1 at a time with a wait time of 1 minute between starting the slice and checking if it is done
  4. So best case is 1 slice per 2 minutes => 240 minutes = 4 hours total run time
  5. Based on some internal slack thread we tracked the similar notes migration we did in the past and it took 9 hours to run. There are less MRs in the index so we should expect this to take less than 9 hours.

The code in this MR is almost identical to what we did for notes in !53013 (merged) including the small fix made in !57248 (merged) .

Screenshots (strongly suggested)

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • Label as security and @ mention @gitlab-com/gl-security/appsec
  • The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • Security reports checked/validated by a reviewer from the AppSec team

Related to #322800 (closed)

Edited by Dylan Griffith

Merge request reports