Skip to content

Populate commit permissions migration taking too long

Related #344459 (closed)

The PopulateCommitPermissionsInMainIndex advanced search migration introduced in !78562 (merged) is taking too long to complete on GitLab.com

Current state:

  • migration has 473397158 records left to process
  • migration processed 288409 documents in ~70 hours. 4120 documents/hour
  • it will take 4700 days to complete
  • MigrationWorker is paused using the ops feature flag elastic_migration_worker
We also saw performance impacts for search latency

image__4_

image__5_

Example log from the Elastic Cloud showing BulkByScrollResponse taking > 6 minutes and often > 10 minutes to complete 200,000 records

16912755 finished with response BulkByScrollResponse[took=6.2m,timed_out=false,sliceId=null,updated=200000,created=0,deleted=0,batches=200,versionConflicts=0,noops=0,retries=0,throttledUntil=0s,bulk_failures=[],search_failures=[]]
Edited by Terri Chu