Skip to content

Batched background migration to prune stale export jobs

Carla Drago requested to merge 383337-bg-prune-stale-export-jobs into master

What does this MR do and why?

This change adds a batched background migration to remove stale project_export_jobs and associated relations. Stale jobs are those older than 7 days.

This batched migration is required as a first step to pruning stale jobs regularly with a cron job (see !102659 (merged)). Since there are millions of stale project_export_jobs, the initial pruning is best handled by a migration like this. We have around 9.6 million records in production and we want to trim 9.1 million out of those, that means we're going to delete around 94% of the records in the project_export_jobs table. Once run, the cron job changes can be deployed safely without risk of degrading database performance.

The associated relations, project_relation_exports and project_relation_export_uploads, have tables that are not currently being used. Specs nevertheless cover these relations as a precaution in the event that some stray records exist.

Please make sure this migration is finalized before releasing the cron job. Given this project_export_jobs is a large table, we may need to span these changes across multiple release to allow for the post-deploy migration execution in self-managed. Example:

  • 15.7 - We ship this background migration
  • 15.8 - Migration is finalized and the cron job is released.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #383337 (closed) Blocks !102659 (merged)

Edited by Carla Drago

Merge request reports