Rollout Sidekiq job payload compression and size limiter

After #1054 (closed) is closed and related MRs are merged, we should start rolling out this features in a safety fashion.

Metrics to observe

• Logs: Sidekiq logs in Kibana

• Metric: Sidekiq Service RPS: https://dashboards.gitlab.net/d/sidekiq-main/sidekiq-overview?viewPanel=975161318&orgId=1 Hide charts

• Metric: Web/API/Git/Sidekiq Error Ratio: https://dashboards.gitlab.net/d/web-main/web-overview?viewPanel=1806708210&orgId=1 / https://dashboards.gitlab.net/d/api-main/api-overview?viewPanel=2019568690&orgId=1 / https://dashboards.gitlab.net/d/git-main/git-overview?viewPanel=1941771315&orgId=1 / https://dashboards.gitlab.net/d/sidekiq-main/sidekiq-overview?viewPanel=4245085395&orgId=1

For each rolling out stage, validate by triggering the workers mentioned in #1054 (comment 567642741) with different scenarios:

• Compress mode: job size is less than compression threshold
• Compress mode: job size is more than the compression threshold but less than the limit after compression
• Compress mode: job size is more than the limit after compression

In each scenario, ensure that the jobs are processed properly (with or without compression), there aren't any errors in the Sidekiq clients. If a job is compressed, compressed and original_job_size_bytes fields are added to the logs.

✅ Rollout to Staging

Related MR:

• production#4708

Set the following environment variables to Staging (both VMs and K8s)

GITLAB_SIDEKIQ_SIZE_LIMITER_MODE=compress 
GITLAB_SIDEKIQ_SIZE_LIMITER_LIMIT_BYTES=5000000 # 5MB
GITLAB_SIDEKIQ_SIZE_LIMITER_COMPRESSION_THRESHOLD_BYTES=100000 # 100kb

✅ Rollout to Production, set the limit size to highest observed payload size

After rolling out the following environment variables, the oversized jobs will be compressed before pushing to Redis. As the limit size is exceptionally high (50MB), no jobs will be discarded.

GITLAB_SIDEKIQ_SIZE_LIMITER_MODE=compress 
GITLAB_SIDEKIQ_SIZE_LIMITER_LIMIT_BYTES=50000000 # 50MB
GITLAB_SIDEKIQ_SIZE_LIMITER_COMPRESSION_THRESHOLD_BYTES=100000 # 100kb

✅ Notify users that will be affected when this is used in enforcing mode

We need to notify stage groups that will be affected before this is used to reject jobs in production. This gives them notice that something is changing and the communication needs to describe what they need to do. See the blocking issue below.

✅ Decrease the size limit on production to the desired value

Finally, rollout the following environment variables to production. After reaching this state, any jobs greater than 5MB after compression will be rejected.

GITLAB_SIDEKIQ_SIZE_LIMITER_MODE=compress 
GITLAB_SIDEKIQ_SIZE_LIMITER_LIMIT_BYTES=5000000 # 5MB
GITLAB_SIDEKIQ_SIZE_LIMITER_COMPRESSION_THRESHOLD_BYTES=100000 # 100kb