Review heavy usages of main primary DB from Sidekiq workers

Screenshot_2024-08-26_at_12.57.01_PM

https://log.gprd.gitlab.net/app/r/s/EAz8Y

Questions

  1. Can we shift some of these workers to replicas ?
  2. Can we shift some of these workers to run at different times (not peak) ? => https://gitlab.com/gitlab-org/gitlab/-/issues/496254
  3. Why does workers that seem be related to CI (PipelineProcessWorker, Ci::BuildFinishedWorker, Ci::InitialPipelineProcessWorker) use so much main DB, when it should be using ci DB ?
  4. What is FlushCounterIncrementsWorker, does it have to run so often, at peak ?
Edited by Thong Kuah