Sam Word authored 1 year ago and Tiger Watson committed 1 year ago

In PruneExpiredExportJobsService, EachBatch won't scale for large
volumes of data because we're trying to batch by pkey and a timestamp.
This introduces a new index on project_export_jobs to use loop-based
deletion and keyset pagination to more efficiently prune old uploads.

Created a new service to handle pruning old ProjectExportService and
its associations efficiently. Specifically, upload files need and
be handled and pruned as well. Before, only associated records were
deleted via an ON DELETE CASCADE clause on their postgres tables.

Carla Drago authored 2 years ago and Alex Ives committed 2 years ago

This change adds a worker to prune ProjectExportJob
records that are older than 7 days. This will
ensure the project_export_jobs DB table remains
a reasonable size. It currently contains millions
of rows.

Changelog: performance

Rodrigo Tomonari authored 2 years ago and Pavel Shutsin committed 2 years ago

In order to parallelize the project export in Import/Export, two new
are being added to Import/Export to track and store the partial
export files.

The table `project_export_relations` will hold the status of the
relation exportation. And the table `project_export_relation_uploads`
will store the location of the exported file.

Changelog: added

The plan for RSpec 4.0 is to disable monkey patching, reference:
https://rspec.info/blog/2013/07/the-plan-for-rspec-3/#zero-monkey-patching-mode
https://relishapp.com/rspec/rspec-core/docs/configuration/zero-monkey-patching-mode

This commit stops using RSpec monkey patching in several specs.

Aishwarya Subramanian authored 5 years ago and Stan Hu committed 4 years ago

Currently, lock files are used to determine the state of a
project export. However, the lock files are not stored in a shared
volume, causing the export states to be reported incorrectly.

In order to fix this, we now store the export states in database
as a more reliable mechanism. The table used is project_export_jobs.
It stores the project_id, job_id and the current state.

In addition to existing states, also added are two new states
queued and regeneration_in_progress states. They are used to indicate
a job that's waiting to be started and the case when a project export
operation has been requested to be re-generated.

Failed jobs are re-tried 3 times, after which the state is updated
to failed.

To account for jobs that get stuck, a cron job runs every hour
to set the stuck jobs to failed state.

Closes #32203