Implement background task to recover throttled worker

What does this MR do and why?

Implement background task to recover throttled worker

When a worker is being throttled, this background task will recover the concurrency limit gradually every minute.

This recovers workers being throttled from throttling middleware !197226 (merged)

References

Second task of gitlab-com/gl-infra/observability/team#3815 (closed)

Screenshots or screen recordings

Before	After

How to set up and validate locally

1. Enable sidekiq_throttling_middleware feature flag
2. On Rails console:

throttler = Gitlab::SidekiqMiddleware::Throttling::Middleware.new(Chaos::DbSleepWorker)
hard_throttle = Gitlab::SidekiqMiddleware::Throttling::Strategy::HardThrottle

throttler.send(:current_limit) # current limit should be 2500

throttler.send(:throttle!, hard_throttle) # throttle the worker

throttler.send(:current_limit) # current limit should be halved at 1250 now

3. Sidekiq logs should show concurrency limit to gradually increase by 10% at roughly every minute

$ rg 'recovery' sidekiq.log
217391:{"severity":"INFO","time":"2025-07-23T15:13:50.865Z","message":"Recovering concurrency limit for Chaos::DbSleepWorker","recovery_strategy":"GradualRecovery","class":"Chaos::DbSleepWorker","previous_concurrency_limit":1250,"new_concurrency_limit":1375,"max_concurrency_limit":2500,"retry":0}
217401:{"severity":"INFO","time":"2025-07-23T15:14:15.474Z","message":"Recovering concurrency limit for Chaos::DbSleepWorker","recovery_strategy":"GradualRecovery","class":"Chaos::DbSleepWorker","previous_concurrency_limit":1375,"new_concurrency_limit":1512,"max_concurrency_limit":2500,"retry":0}
217413:{"severity":"INFO","time":"2025-07-23T15:15:23.196Z","message":"Recovering concurrency limit for Chaos::DbSleepWorker","recovery_strategy":"GradualRecovery","class":"Chaos::DbSleepWorker","previous_concurrency_limit":1512,"new_concurrency_limit":1663,"max_concurrency_limit":2500,"retry":0}
217423:{"severity":"INFO","time":"2025-07-23T15:16:33.414Z","message":"Recovering concurrency limit for Chaos::DbSleepWorker","recovery_strategy":"GradualRecovery","class":"Chaos::DbSleepWorker","previous_concurrency_limit":1663,"new_concurrency_limit":1829,"max_concurrency_limit":2500,"retry":0}
217433:{"severity":"INFO","time":"2025-07-23T15:17:53.949Z","message":"Recovering concurrency limit for Chaos::DbSleepWorker","recovery_strategy":"GradualRecovery","class":"Chaos::DbSleepWorker","previous_concurrency_limit":1829,"new_concurrency_limit":2011,"max_concurrency_limit":2500,"retry":0}

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.