Reschedule Delete Orphaned Deployments BG migration (!66813) · Merge requests · GitLab.org / GitLab

Shinya Maeda requested to merge reschedule-delete-orphaned-deployments into master Jul 26, 2021

What does this MR do?

We shipped a background migration in the previous milestone, but it seems approx. 5% of batches/jobs failed on gitlab.com due to statement timeout. We'll reschedule the migration with a smaller batch size that unlikely hits the same error.

BG migration estimation

batch size = 10,000 (Previously, 100,000)
Total batch count = 163,863,216 / 10,000 = 16,390 jobs
Estimated total timing: 16,390 batches * 2 min per batch = 22.76 days to finish executing all the scheduled jobs in Sidekiq.

The failure details of the initial background migration

Reference: https://log.gprd.gitlab.net/goto/5183cb0ccca81128d076c2f2f6ecf12b

Pending batches

[ gprd ] production> Gitlab::Database::BackgroundMigrationJob.where(class_name: 'DeleteOrphanedDeployment
s').pending.count
=> 80

8_000_000 million rows

Pending batch details

Gitlab::Database::BackgroundMigrationJob.where(class_name: 'DeleteOrphanedDeployments').order(:id).pending.each do |job|
  puts "Batch Id range: from: #{job.arguments.first} to: #{job.arguments.second}"
end

Batch Id range: from: 7002960 to: 7110854
Batch Id range: from: 10135691 to: 10238873
Batch Id range: from: 13567241 to: 13678031
Batch Id range: from: 14305566 to: 14409857
Batch Id range: from: 16912890 to: 17015652
Batch Id range: from: 17537017 to: 17641686
Batch Id range: from: 21184360 to: 21287323
Batch Id range: from: 23579994 to: 23684931
Batch Id range: from: 29654599 to: 29763185
Batch Id range: from: 30408513 to: 30513361
Batch Id range: from: 32604748 to: 32712560
Batch Id range: from: 36044245 to: 36147250
Batch Id range: from: 39264255 to: 39369030
Batch Id range: from: 39369031 to: 39471253
Batch Id range: from: 43277091 to: 43379342
Batch Id range: from: 43996239 to: 44098760
Batch Id range: from: 46763148 to: 46865166
Batch Id range: from: 48710213 to: 48812514
Batch Id range: from: 49017348 to: 49120024
Batch Id range: from: 51684975 to: 51788433
Batch Id range: from: 53122372 to: 53224467
Batch Id range: from: 55174664 to: 55276844
Batch Id range: from: 55788198 to: 55890801
Batch Id range: from: 60079701 to: 60181173
Batch Id range: from: 60181174 to: 60282283
Batch Id range: from: 63733134 to: 63834470
Batch Id range: from: 64948646 to: 65050266
Batch Id range: from: 68693368 to: 68794483
Batch Id range: from: 71629756 to: 71730945
Batch Id range: from: 73651686 to: 73753040
Batch Id range: from: 80016244 to: 80117241
Batch Id range: from: 81026632 to: 81127221
Batch Id range: from: 84865735 to: 84966309
Batch Id range: from: 86682478 to: 86783037
Batch Id range: from: 87186790 to: 87287482
Batch Id range: from: 89713684 to: 89817159
Batch Id range: from: 90828017 to: 90929278
Batch Id range: from: 92948686 to: 93050968
Batch Id range: from: 93252708 to: 93354073
Batch Id range: from: 97491662 to: 97592229
Batch Id range: from: 97592230 to: 97692903
Batch Id range: from: 98197458 to: 98297964
Batch Id range: from: 103033965 to: 103134368
Batch Id range: from: 104947810 to: 105048852
Batch Id range: from: 105754182 to: 105854881
Batch Id range: from: 107465749 to: 107566837
Batch Id range: from: 109478652 to: 109579171
Batch Id range: from: 111491963 to: 111592404
Batch Id range: from: 113404731 to: 113505346
Batch Id range: from: 115215086 to: 115315593
Batch Id range: from: 119339438 to: 119440095
Batch Id range: from: 121154654 to: 121254970
Batch Id range: from: 123567114 to: 123667532
Batch Id range: from: 126180343 to: 126281202
Batch Id range: from: 127386951 to: 127487443
Batch Id range: from: 133612688 to: 133713145
Batch Id range: from: 135522462 to: 135622892
Batch Id range: from: 135622893 to: 135723500
Batch Id range: from: 138946179 to: 139046754
Batch Id range: from: 139046755 to: 139147493
Batch Id range: from: 139147494 to: 139247869
Batch Id range: from: 139247870 to: 139348335
Batch Id range: from: 139348336 to: 139448836
Batch Id range: from: 139750370 to: 139850942
Batch Id range: from: 140253192 to: 140354168
Batch Id range: from: 141159694 to: 141260546
Batch Id range: from: 143975967 to: 144076335
Batch Id range: from: 145984348 to: 146084773
Batch Id range: from: 152915238 to: 153016312
Batch Id range: from: 154122737 to: 154223051
Batch Id range: from: 155930920 to: 156031974
Batch Id range: from: 157036732 to: 157137089
Batch Id range: from: 158541265 to: 158641509
Batch Id range: from: 160447814 to: 160547976
Batch Id range: from: 163562107 to: 163662793
Batch Id range: from: 163662794 to: 163763261
Batch Id range: from: 163763262 to: 163863579
Batch Id range: from: 164565111 to: 164665340
Batch Id range: from: 164665341 to: 164765772
Batch Id range: from: 167673477 to: 167773767

Related #335071 (closed)

Migration on local environment

shinya@shinya-B550-VISION-D:~/workspace/thin-gdk/services/rails/src$ tre bin/rails db:migrate:redo VERSION=20210722010101
INFO: This script is a predefined script in devkitkat.
== 20210722010101 RescheduleDeleteOrphanedDeployments: reverting ==============
== 20210722010101 RescheduleDeleteOrphanedDeployments: reverted (0.0000s) =====

== 20210722010101 RescheduleDeleteOrphanedDeployments: migrating ==============
-- Scheduled 1 DeleteOrphanedDeployments jobs with a maximum of 1000 records per batch and an interval of 120 seconds.

The migration is expected to take at least 120 seconds. Expect all jobs to have completed after 2021-07-26 07:16:32 UTC."
== 20210722010101 RescheduleDeleteOrphanedDeployments: migrated (0.0631s) =====

Does this MR meet the acceptance criteria?

Conformity

I have included changelog trailers, or none are needed. (Does this MR need a changelog?)
I have added/updated documentation, or it's not needed. (Is documentation required?)
I have properly separated EE content from FOSS, or this MR is FOSS only. (Where should EE code go?)
I have added information for database reviewers in the MR description, or it's not needed. (Does this MR have database related changes?)
I have self-reviewed this MR per code review guidelines.
This MR does not harm performance, or I have asked a reviewer to help assess the performance impact. (Merge request performance guidelines)
I have followed the style guides.
This change is backwards compatible across updates, or this does not apply.

Availability and Testing

I have added/updated tests following the Testing Guide, or it's not needed. (Consider all test levels. See the Test Planning Process.)
I have tested this MR in all supported browsers, or it's not needed.
I have informed the Infrastructure department of a default or new setting change per definition of done, or it's not needed.

Security

Does this MR contain changes to processing or storing of credentials or tokens, authorization and authentication methods or other items described in the security review guidelines? If not, then delete this Security section.

Label as security and @ mention @gitlab-com/gl-security/appsec
The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
Security reports checked/validated by a reviewer from the AppSec team

Edited Jul 28, 2021 by Shinya Maeda

Reschedule Delete Orphaned Deployments BG migration