Handle stopping of background jobs in maintenance/read-only mode
We want to handle the shutting down of Sidekiq during maintenance mode, since everything will be read only.
From our disaster recovery docs this is what we suggest:
Disable non-Geo periodic background jobs on the primary node by navigating to Admin Area > Monitoring > Background Jobs > Cron, pressing Disable All, and then pressing Enable for the geo_sidekiq_cron_config_worker cron job. This job will re-enable several other cron jobs that are essential for planned failover to complete successfully.
We need to automate as much of it as we can, and inform the user when they set maintenance mode.
This issue needs some research on what we do with on-going jobs and how and whether to drain the queues before starting maintenance mode.