Sidekiq metrics exporter metrics are broken

It looks like !81983 (merged) broke metrics we export about the Sidekiq metrics exporter process:

Screenshot_from_2022-03-22_10-45-00

I think the problem here is:

The problem is that Sidekiq executes things in a different order than Puma, because sidekiq-cluster is not a Rails process, and it does not know when workers are fully started or how to communicate with them at all (outside of sending signals.)

To fix this, we should consider:

  1. Not wiping metrics in the initializer for Sidekiq since that again arbitrarily executes logic in only one Sidekiq process on behalf of all others, which leads to race conditions. It is OK for Puma because with Puma, there is a dedicated primary that starts before any workers do.
  2. Moving this logic into the sidekiq-cluster parent script (cli.rb), before either the metrics server or any worker are started.
Edited by Matthias Käppler