Skip to content

Backfill LfsObjectsProject records of forks

Patrick Bajao requested to merge 55487-backfill-lfs-objects-v2 into master

What does this MR do?

LinkLfsObjects background migration jobs were disabled as it's causing some statement timeouts on staging.

This re-schedules the background migration jobs but this time it'll not result to a statement timeout.

Related MRs:

Migration

== 20200217091401 RescheduleLinkLfsObjects: migrating =========================
== 20200217091401 RescheduleLinkLfsObjects: migrated (0.2321s) ================

Ran this test on #database-lab: https://gitlab.com/snippets/1942110. Created 1k forks of a source project with 100k LFS objects. The last 2 queries in that snippet are for querying the forks in Gitlab::BackgroundMigration::LinkLfsObjects and the insert query.

Given the query plans of those queries a background migration job can finish in around 22 seconds. Since we have 14,094 forks to be migrated as of this writing, that means we will have 15 background migration jobs enqueued with 2 minute delay. That said, this means the migration will be done in ~30 minutes.

This is based on the test data that I created. The formula I used is:

((18 + ((13 * 100) * 1000)) / 1000) / 60

Where:

  • 18 - in ms, time spent on querying the forks (1k per background migration job)
  • 13 - in ms, time spent on inserting 1k LfsObjectsProject records
  • 100 - number of insert batches (100k records per project, 1k batch per insert)
  • 1000 - number of forks to be migrated per background migration job
  • 1000 - ms in seconds
  • 60 - seconds in minutes

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • Label as security and @ mention @gitlab-com/gl-security/appsec
  • The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • Security reports checked/validated by a reviewer from the AppSec team

#55487 (closed)

Edited by 🤖 GitLab Bot 🤖

Merge request reports