Defer Direct Transfer workers in case DB is unhealthy
Update PipelineWorker and PipelineBatchWorker workers to use the
defer_on_database_health_signal
option.
Also, update defer_on_database_health_signal to receive a block to lazy evaluate the schema and table as the workers modify different tables depending on the relation being imported
Bear in mind that some pipelines might update several tables as they can save nested resources. For example, MergeRequestsPipeline saves records in the merge_requests
and notes
table. However, this change only checks the autovacuum
in the parent table.
Certain pipelines may update multiple tables, especially when they handle nested resources. For instance, the MergeRequestsPipeline can store records in both the merge_requests
and notes
tables. However, for simplicity, this modification only checks for autovacuum
status in the parent/main table.
Related to: #429871
Screenshots or screen recordings
Screenshots are required for UI changes, and strongly recommended for all other merge requests.
Before | After |
---|---|
How to set up and validate locally
- Enable Direct Transfer following these steps
- Enable the feature flags
bulk_import_deferred_workers
andbatched_migrations_health_status_autovacuum
- Modify the Gitlab::Database::HealthStatus::Indicators::AutovacuumActiveOnTable file to simulate the
autovacuum
. Follow an example:
diff --git a/lib/gitlab/database/health_status/indicators/autovacuum_active_on_table.rb b/lib/gitlab/database/health_status/indicators/autovacuum_active_on_table.rb
index 6bf2bbf0c707..39f7dbfa8252 100644
--- a/lib/gitlab/database/health_status/indicators/autovacuum_active_on_table.rb
+++ b/lib/gitlab/database/health_status/indicators/autovacuum_active_on_table.rb
@@ -30,6 +30,9 @@ def enabled?
end
def active_autovacuums_for(tables)
+ return tables if rand(2) == 1 # defer 50% of workers
+ # return tables if tables.include?('merge_requests') # defer if pipeline updates the merge_requests table
+
Gitlab::Database::PostgresAutovacuumActivity.for_tables(tables)
end
end
- Start Sidekiq
- Perform a Direct Transfer migration
- On Sidekiq Dashboard, check if the workers are being rescheduled to execute in the future or or review the job status in the
sidekiq.log
for any jobs marked asdeferred
.
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.