Separate event workers into separate Sidekiq shard

Problem to solve

  • We intend to scale the usage of Gitlab::EventStore in the coming months.
  • This service is based on Sidekiq and currently funnels jobs into our primary job queue.
  • We risk destabilizing other background processing by consuming the queue with event processing jobs.

Proposal

  • Spin up the new sidekiq shard
  • Spin up the new redis instance
  • Configure GitLab-rails to send jobs to this new redis
  • Migrate the existing event workload over to a new queue on the new redis instance.

References

Edited by Jeff Tucker