Throttling of Sidekiq jobs
Sidekiq jobs are currently unthrottled in any way. This means that a clever set of for example Git pushes could overload a GitLab instance. As mentioned in https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/6802#note_16965325 and the comments leading up to it we need a way to throttle Sidekiq jobs. Some tools that could do this:
- https://github.com/gevans/sidekiq-throttler (unmaintained as per https://github.com/gevans/sidekiq-throttler/issues/26)
- https://github.com/brainopia/sidekiq-limit_fetch
What would be nice to have:
- The ability to set an upper limit to the number of workers. This can be used to prevent a single worker from hogging all threads
- The ability to throttle the number of jobs per minute/hour/etc
- The ability to throttle jobs per user. This allows us to throttle jobs when a malicious user starts pushing a lot of data (for example)
Important: whatever we ship should by default not throttle/limit anything, this system should be opt-in so it doesn't disrupt the service of existing users. Furthermore we should be able to configure things either via Chef, or via the database. This depends a bit on whatever tool we use for throttling.