Limit the size of a Sidekiq job

What does this MR do?

A part of gitlab-com/gl-infra/scalability#825 (closed).

This MR implements a Sidekiq middleware to validate the size of a job payload. This middleware has 2 modes: track mode and raise mode.

• In track mode (default), if the size of the job payload exceeds the limit, an event will be sent to Sentry and the job continues to be scheduled.
• In raise mode, the oversized job will be rejected and an exception will be raised.

This middleware ignores any job from the workers having big_payload! attribute.

The mode and size limit are fetched from the environment variables GITLAB_SIDEKIQ_SIZE_LIMITER_MODE and GITLAB_SIDEKIQ_SIZE_LIMITER_LIMIT_BYTES. By default, the middleware won't take any action. In the plan, we'll execute the following steps after this MR is merged into the production:

• Set a size limit. The desired size limit may be the 99th percentile of the job payload size tracked in logs.
• For each violation, we'll open an issue, maybe in Scalability issue board and tag related stage group in to resolve the situation.
• If a worker is too hard to resolve right now, we can add it into an allowlist with big_payload! attribute
• Enable raise mode in the production
• Finally, we clean up all workers having big_payload! attribute.

Screenshots (strongly suggested)

Exception thrown into Sentry

The exception is logged into the exception log

When the raise mode is enabled, the exception is threw out in the development environment

Warning in sidekiq client logs when the configuration is invalid

Does this MR meet the acceptance criteria?

Conformity

• Changelog entry
• Documentation (if required)
• Code review guidelines
• Merge request performance guidelines
• Style guides
• Database guides
• Separation of EE specific content

Availability and Testing

• Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process.
• Tested in all supported browsers
• Informed Infrastructure department of a default or new setting change, if applicable per definition of done

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

• Label as security and @ mention @gitlab-com/gl-security/appsec
• The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
• Security reports checked/validated by a reviewer from the AppSec team