Skip to content

Defer Direct Transfer workers in case DB is unhealthy

Rodrigo Tomonari requested to merge rodrigo/defer-workers into master

Update PipelineWorker and PipelineBatchWorker workers to use the defer_on_database_health_signal option.

Also, update defer_on_database_health_signal to receive a block to lazy evaluate the schema and table as the workers modify different tables depending on the relation being imported

Bear in mind that some pipelines might update several tables as they can save nested resources. For example, MergeRequestsPipeline saves records in the merge_requests and notes table. However, this change only checks the autovacuum in the parent table.

Certain pipelines may update multiple tables, especially when they handle nested resources. For instance, the MergeRequestsPipeline can store records in both the merge_requests and notes tables. However, for simplicity, this modification only checks for autovacuum status in the parent/main table.

Related to: #429871

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

Before After

How to set up and validate locally

  1. Enable Direct Transfer following these steps
  2. Enable the feature flags bulk_import_deferred_workers and batched_migrations_health_status_autovacuum
  3. Modify the Gitlab::Database::HealthStatus::Indicators::AutovacuumActiveOnTable file to simulate the autovacuum. Follow an example:
diff --git a/lib/gitlab/database/health_status/indicators/autovacuum_active_on_table.rb b/lib/gitlab/database/health_status/indicators/autovacuum_active_on_table.rb
index 6bf2bbf0c707..39f7dbfa8252 100644
--- a/lib/gitlab/database/health_status/indicators/autovacuum_active_on_table.rb
+++ b/lib/gitlab/database/health_status/indicators/autovacuum_active_on_table.rb
@@ -30,6 +30,9 @@ def enabled?
           end
 
           def active_autovacuums_for(tables)
+            return tables if rand(2) == 1 # defer 50% of workers
+            # return tables if tables.include?('merge_requests') # defer if pipeline updates the merge_requests table
+
             Gitlab::Database::PostgresAutovacuumActivity.for_tables(tables)
           end
         end
  1. Start Sidekiq
  2. Perform a Direct Transfer migration
  3. On Sidekiq Dashboard, check if the workers are being rescheduled to execute in the future or or review the job status in the sidekiq.log for any jobs marked as deferred.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Rodrigo Tomonari

Merge request reports