Handle Sidekiq jobs with a compressed payload > 5MB are being rejected
In &490 (closed), we applied a Sidekiq middleware to compress the job payload before pushing into Redis. Any oversized jobs exceeding a threshold on GitLab.com are rejected. At that time, a Gitlab::SidekiqMiddleware::SizeLimiter::ExceedLimitError
exception is raised, and the jobs will never be executed. At the moment, GitLab.com's threshold is 5MB after compression. This number is more than generous in most cases. However, we still have a handful of rejections per week. We don't have a way to handle those jobs automatically. They need human intervention to capture the exceptions and handle each case accordingly. The Scalability team should coordinate with stage groups who are familiar with the matter to tackle the issues.
This issue is a convenient place to keep track of known rejections.
Status | Location | Description | Issue |
---|---|---|---|
EmailReceiverWorker |
If we can't handle an incoming email, we send a rejection email to the sender. The original payload was attached to the mailer. If the raw payload is huge, the rejection mailer is properly rejected. | gitlab-org/gitlab#340167 (closed) | |
WebhookWorker |
When we log the webhooks executions, we try to log the entire response (body + headers) from WebHookService#log_execution without limits | https://gitlab.com/gitlab-org/gitlab/-/issues/340027 | |
Integration |
We've received a push for a project that has integrations turned on. The push resulted in a payload that was bigger than 5MB after compression. So it was rejected. | https://gitlab.com/gitlab-org/gitlab/-/issues/340186 | |
MergeRequests::RefreshService |
A push for a project resulted in an email being generated that was larger than 5MB after compression so it was rejected | gitlab-org/gitlab#351293 (closed) | |
WikiPages::BaseService |
A wiki page was updated for a project that has integrations turned on. The entire content of the wiki-page is included in the payload | gitlab-org/gitlab#367628 (closed) |
The list of rejections can be found at https://log.gprd.gitlab.net/goto/558fbc0dd1e5c53b69f9e95c542b36b1.
Dashboard to track sidekiq workers by 99th percentile of compressed payload as an early warning.