Prioritize Sidekiq queues differently
Based on discussion in #infrastructure chat. Currently, our Sidekiq queue are defined without weight.
start_sidekiq()
{
exec bundle exec sidekiq -q post_receive -q mailers -q archive_repo -q system_hook -q project_web_hook -q gitlab_shell -q incoming_email -q runner -q common -q default -e $RAILS_ENV -P $sidekiq_pidfile "$@"
}
According to https://github.com/mperham/sidekiq/wiki/Advanced-Options#queues, lack of any weight means all queues will be processed linearly. The first queue will be 100% empty before a job is pulled from the second queue, and so on. In high volume systems this can be a concern. No default
or runner
queue jobs will be processed unless all 8-10 other queues are empty.
The other thing I suspect may happen in this case, is certain types of jobs will all run together and that could cause database lock contention. Consider that all CI runner jobs will execute very near to each other. If each of those has to query a database table, that could be a problem. This is a theoretical problem, but seems plausible to me. By adding some weight we can spread out the different type of jobs a little better.
There seem to be multiple ways we can do this.
Weight
Assign an explicit weight to each queue. Higher numbers mean higher 'priority'. For example if you have sidekiq -q critical,2 -q default
, the critical queue will be polled approximately 2 times for every 1 time the default queue is polled. Equal weight means they will be polled equally as often. This makes me think we should have at least weight 1 on all queues. But some queues may benefit from a higher weight. For example, maybe the default weight should be 3, post_receive
should be 4. Maybe incoming_email
should be 1 or 2, etc. I am not sure we can use values like 1.2, etc. Docs don't say anything but only show whole numbers.
Limits
There are also other gems, like sidekiq-limit_fetch
. This gives us the ability to set a limit on the number of active jobs for a given queue. For example, you could restrict incoming email to 1 concurrent job if too many at once have potential to cause contention. limits
affects concurrent jobs among all workers. process_limits
affects number of jobs per process.
queues:
- one_at_a_time_job
- internal_lookup_job
- memory_hog_job
- welcome_email_job
- data_backfill_job
:limits:
one_at_a_time_job: 1
internal_lookup_job: 5
:process_limits:
memory_hog_job: 1