Test migrations against background scheduling logic
Recently a migration failed and it is suspected that the scheduling logic may have been a part of the problem gitlab-com/gl-infra/production#2802 (comment 425449476)
Can we test scheduling logic prior to pushing to production? Is this a pervasive problem?
Designs
- Show closed items
Activity
-
Newest first Oldest first
-
Show all activity Show comments only Show history only
- Craig Gomes added to epic &3 (closed)
added to epic &3 (closed)
- Craig Gomes changed the description
Compare with previous version changed the description
- Andreas Brandl changed epic to &6 (closed)
changed epic to &6 (closed)
- Andreas Brandl mentioned in merge request gitlab-org/gitlab!47723 (merged)
mentioned in merge request gitlab-org/gitlab!47723 (merged)
- Developer
Here is an example where the scheduling went wrong: gitlab-org/gitlab!47723 (comment 487605436)
An idea is to implement background migration jobs more declaratively, so that we can reason about which jobs to expect. Next, we'd run the scheduling on production data and add tests to check what exactly ends up in the Redis queue. This way, we don't need to run sidekiq.
- Andreas Brandl added database groupdatabase labels
added database groupdatabase labels
- Maintainer
Setting label(s) Category:Database ~"devops::enablement" ~"section::enablement" based on groupdatabase.
- 🤖 GitLab Bot 🤖 added Category:Database devopssystems sectioncore platform labels
added Category:Database devopssystems sectioncore platform labels
Here's another example where I missed a
*
in the background migration arguments: gitlab-org/gitlab!51475 (merged)I'm currently trying out gitlab-org/gitlab!54339 (merged) to verify the number of arguments in specs. The problem is that we don't always have specs for post-deployment migrations, so this would have caught gitlab-org/gitlab!51475 (merged) but not gitlab-org/gitlab!47723 (merged).
Another option I considered is adding checks in
BackgroundMigrationHelpers
before actually queuing jobs, but this seems brittle and again relies on either having specs, or the correct records in the DB when running it in development.- Markus Koller mentioned in merge request gitlab-org/gitlab!54339 (merged)
mentioned in merge request gitlab-org/gitlab!54339 (merged)
- Chun Du added Engineering Allocation label
added Engineering Allocation label
- Andreas Brandl changed epic to &8 (closed)
changed epic to &8 (closed)
- Craig Gomes set weight to 1
set weight to 1
- Alex Ives mentioned in issue #180 (closed)
mentioned in issue #180 (closed)
- Andreas Brandl mentioned in issue gitlab-com/gl-infra/production#4843 (closed)
mentioned in issue gitlab-com/gl-infra/production#4843 (closed)
- Craig Gomes changed epic to &10
changed epic to &10
- Simon Tomlinson added databaseactive workflowready for development labels
added databaseactive workflowready for development labels
- Alex Ives mentioned in issue gitlab-org/gitlab#347673 (closed)
mentioned in issue gitlab-org/gitlab#347673 (closed)
- Developer
Closing gitlab-org/gitlab#347673 (closed) and moving here.
We may use an opt-in signal to tell we want to run the background workers inline. It can be an environment variable, a label in the merge request, or some text in the merge request title.
Override queue_background_migration_jobs_by_range_at_intervals - migrate_in to not schedule a background worker, but run the actual queries during the background migrations
- Mayra Cabrera mentioned in merge request gitlab-org/gitlab!77103 (closed)
mentioned in merge request gitlab-org/gitlab!77103 (closed)
- Yannis Roussos mentioned in issue #242 (closed)
mentioned in issue #242 (closed)
- 🤖 GitLab Bot 🤖 added devopsdata stores label and removed devopssystems label
added devopsdata stores label and removed devopssystems label
@stomlinson @dfrazao-gitlab is this something that's still needed with the shift to bbm and the current state of the migration testing?
Collapse replies - Maintainer
@alexives we run the scheduling logic for batched background migrations, but don't fail the pipeline if it fails, or report on the time taken per-query while running scheduling. I think this would be a great feature addition.
2
- Alex Ives added typefeature label and removed Engineering Allocation label
added typefeature label and removed Engineering Allocation label
- 🤖 GitLab Bot 🤖 added groupdatabase frameworks label and removed groupdatabase [DEPRECATED] label
added groupdatabase frameworks label and removed groupdatabase [DEPRECATED] label