Handle cronjobs in zonal sidekiq clusters environment
The Sidekiq-Cron engine in GitLab codebase is not compatible with the zonal sidekiq clusters environment. It's working flow is analyzed in details in this comment. In a nutshell, the engine uses a set of per-cronjob keys to log the latest execution time. These keys act as a global lock ensuring only one worker executes a cron job at a right time. When there are multiple Redis instances, the workers can't synchronize. It's likely the cron jobs are duplicated for each cluster.
A good long-term solution is to use a shared redis instance. This solution is not feasible at the moment. sidekiq-cron doesn't allow us to switch the redis pool. It uses Sidekiq.redis
underlying. We can contribute to add a redis pool option, but the reaction of the maintainers is not guaranteed.
An easier solution for us at the moment is to enable the cron jobs in one cluster only. Other clusters disable the cron job functionality at load time. This solution is dead simple. It may create workload imbalance. This weakness may not be a show-stopping problem at the moment.
Apart from the built-in cronjob, there are two components that also manage cronjob system: Gitlab::Mirror and Gitlab::Geo::CronManager. Those components toggles the cron jobs accordingly to the roles (primary/secondary Geo role) of the node the process is running on. We should ask for consultation from groupgeo.