Evaluate current usages of `data_consistency: always`

There are many workers today with data_consistency: always. Any query that is read-only may be run on primary when it could be run in replica

Action items

  • Start with a list of sidekiq workers assembled from the marginalia comments in the top 15 queries by total_time ordered by frequency
  • Evaluate inline disable of the SidekiqLoadBalancing/WorkerDataConsistency to make sure it is still true. Add comments to the rubocop:disable. Shift to using :delayed or :sticky where possible.
  • Work though all workers in .rubocop_todo/sidekiq_load_balancing/worker_data_consistency.yml. Shift to using :delayed or :sticky` where possible, or inline disable the cop.
  • Create a followup issue to write a Gitlab Housekeeper Keep that automates changing workers from always to sticky unless the team responsible has a concrete reason it cannot be supported, and has added an inline disable comment with a reason and a followup issue linked to remediate the worker to make it possible to change to sticky.
Edited by Leonardo da Rosa