Skip to content
Snippets Groups Projects
Commit d1d75b58 authored by Prabakaran Murugesan's avatar Prabakaran Murugesan :two: Committed by Amy Qualls
Browse files

Documentation for deferring sidekiq using db health status

MR: !119190

Changelog: other
parent b14510be
No related branches found
No related tags found
2 merge requests!119439Draft: Prevent file variable content expansion in downstream pipeline,!119190Doc changes for throttling sidekiq workers based on Db health
......@@ -280,6 +280,21 @@ class CopyColumnUsingBackgroundMigrationJob < BatchedMigrationJob
end
```
## Throttling batched migrations
Because batched migrations are update heavy and there were few incidents in the past because of the heavy load from migrations while the database was underperforming, a throttling mechanism exists to mitigate them.
These database indicators are checked to throttle a migration. On getting a
stop signal, the migration is paused for a set time (10 minutes):
- WAL queue pending archival crossing a threshold.
- Active autovacuum on the tables on which the migration works on.
- Patroni apdex SLI dropping below the SLO.
It's an ongoing effort to add more indicators to further enhance the
database health check framework. For more details, see
[epic 7594](https://gitlab.com/groups/gitlab-org/-/epics/7594).
## Additional filters
By default, when creating background jobs to perform the migration, batched background migrations
......
......@@ -114,6 +114,41 @@ def perform(project_id)
end
```
## Deferring Sidekiq workers
Sidekiq workers are deferred by two ways,
1. Manual: Feature flags can be used to explicitly defer a particular worker, more details can be found [here](../feature_flags/index.md#deferring-sidekiq-jobs).
1. Automatic: Similar to the [throttling mechanism](../database/batched_background_migrations.md#throttling-batched-migrations) in batched migrations, database health indicators are used to defer a Sidekiq worker.
To use the automatic deferring mechanism, worker has to opt-in by calling `defer_on_database_health_signal` with gitlab_schema, delay_by (time to delay) and tables (which is used by autovacuum db indicator) as it's parameters.
**Example:**
```ruby
module Chaos
class SleepWorker # rubocop:disable Scalability/IdempotentWorker
include ApplicationWorker
data_consistency :always
sidekiq_options retry: 3
include ChaosQueue
defer_on_database_health_signal :gitlab_main, 1.minute, [:users]
def perform(duration_s)
Gitlab::Chaos.sleep(duration_s)
end
end
end
```
For deferred jobs, logs contain the following to indicate the source:
- `job_status`: `deferred`
- `job_deferred_by`: 'feature_flag' or 'database_health_check'
## Sidekiq Queues
Previously, each worker had its own queue, which was automatically set based on the
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment