Prometheus client "too many outstanding requests" error when webhook worker is deferred

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

• Label this issue
• Close this issue

Problem

When the new MergeRequests::Refresh::WebHooksWorker introduced in !203215 (merged) is deferred due to database health signals, we encounter the following error:

Gitlab::PrometheusClient::UnexpectedResponseError

429 - {"status":"error","errorType":"too_many_requests","error":"too many outstanding requests"} (Gitlab::PrometheusClient::UnexpectedResponseError)

Background

The webhook worker was introduced as part of splitting webhook logic from MergeRequests::RefreshService and includes database health signal deferral:

defer_on_database_health_signal :gitlab_main, [], 10.seconds

When the worker is deferred (requeued) due to autovacuum activity on the merge_requests table, it appears to be causing too many concurrent requests to the Prometheus client, resulting in rate limiting. We tried to remove the merge_requests table from the array but it didn't work.

https://new-sentry.gitlab.net/organizations/gitlab/issues/1951962/?query=start%3D2025-09-23T00%3A34%3A00%26end%3D2025-09-24T01%3A42%3A00%26groupStatsPeriod%3Dauto&referrer=alerts-related-issues-issue-stream

https://log.gprd.gitlab.net/app/r/s/gW8t2