Blackhole for Sidekiq jobs
In &1004 we've added a way to pause Sidekiq jobs for a specific worker. It proved very handy in production#14758 to prevent load on a database caused by a specific job.
The fix was deployed in a couple of days. In the meantime, we were collecting & deferring jobs until we had over 2 million of them. At this point, after the fix was deployed. We could not disable the deferring of jobs, because that would cause a thundering herd of Sidekiq jobs with further consequences downstream. In this case, we know that the jobs were not valuable any more, so we worked around that by moving the jobs to a separate queue that was unwatched, and then dropped that queue. This required several config changes and coordination between enabling & disabling the jobs.
This would have been easier if we had a similar feature flag that we could toggle that would not defer the jobs, but drop them all together. When this feature flag is enabled, the job would not execute when picked up by Sidekiq, but instead log a message with job_status: "dropped".