Use Sidekiq Cluster by default for GDK (re-add)
What does this Merge Request do?
That's a "revert" from !1127 (merged). I initially saw a few timeouts when trying to start Sidekiq through GDK, but these were unrelated to this change. I had a few Sidekiq processes running on my machine, which apparently led to the issue.
Original description:
We're moving towards making Sidekiq Cluster the new default for every installation, and that includes GDK (see gitlab-com/gl-infra&181 (closed)).
Currently Sidekiq Cluster is used at GitLab.com and big EE customers' scale meaning that putting it in front of developers can prevent issues from flowing to other environments (specially if we run multiple processes on GDK in the future).
With this change we're making Sidekiq Cluster spin a single Sidekiq process by default through the bin/background_jobs script. Mapping the same behavior it used to have while running sidekiq directly.
Removing the SIDEKIQ_WORKERS env var will make it fall back to the old bundle exec sidekiq behavior (if any issue arise).
This MR is specially related to gitlab!27042 (merged).
Testing
2020-03-17_19:06:54.29553 rails-background-jobs : {"severity":"INFO","time":"2020-03-17T19:06:54.295Z","message":"Starting cluster with 1 processes"}
2020-03-17_19:06:53.02899 webpack : ℹ ｢wds｣: Content not from webpack is served from /Users/osw/projects/gitlab/gdk-ee/gitlab
2020-03-17_19:06:53.36496 rails-background-jobs : 2020-03-17T19:06:53.364Z 3244 TID-ouojot0o4 INFO: Bye!
2020-03-17_19:06:53.74385 rails-background-jobs : {"severity":"INFO","time":"2020-03-17T19:06:53.743Z","message":"A worker terminated, shutting down the cluster"}
2020-03-17_19:06:54.29553 rails-background-jobs : {"severity":"INFO","time":"2020-03-17T19:06:54.295Z","message":"Starting cluster with 1 processes"}2020-03-17_19:07:07.13062 rails-background-jobs : 2020-03-17T19:07:07.130Z 6160 TID-oukecdgds INFO: Booting Sidekiq 5.2.7 with redis options {:path=>"/Users/osw/projects/gitlab/gdk-ee/redis/redis.socket", :namespace=>"resque:gitlab", :id=>"Sidekiq-server-PID-6160", :url=>nil}
2020-03-17_19:07:07.16245 rails-background-jobs : 2020-03-17T19:07:07.159Z 6160 TID-oukecdgds INFO: GitLab reliable fetch activated!
2020-03-17_19:07:07.81389 rails-background-jobs : 2020-03-17T19:07:07.813Z 6160 TID-oukecdgds INFO: Cron Jobs - add job with name: stuck_ci_jobs_worker
2020-03-17_19:07:07.82763 rails-background-jobs : 2020-03-17T19:07:07.827Z 6160 TID-oukecdgds INFO: Cron Jobs - add job with name: pipeline_schedule_worker
2020-03-17_19:07:07.83836 rails-background-jobs : 2020-03-17T19:07:07.838Z 6160 TID-oukecdgds INFO: Cron Jobs - add job with name: expire_build_artifacts_worker
... Removed for shortness
name: geo_secondary_registry_consistency_worker
2020-03-17_19:07:42.17991 webpack : 258 entries from '/pages' automatically added to webpack output.
2020-03-17_19:07:45.42279 webpack : ℹ ｢wdm｣: Compiled with warnings.
2020-03-17_19:07:52.07851 rails-background-jobs : 2020-03-17T19:07:52.078Z 6160 TID-oukecdgds INFO: Running in ruby 2.6.5p114 (2019-10-01 revision 67812) [x86_64-darwin18]
2020-03-17_19:07:52.07856 rails-background-jobs : 2020-03-17T19:07:52.078Z 6160 TID-oukecdgds INFO: See LICENSE and the LGPL-3.0 for licensing details.
2020-03-17_19:07:52.07858 rails-background-jobs : 2020-03-17T19:07:52.078Z 6160 TID-oukecdgds INFO: Upgrade to Sidekiq Pro for more features and support: http://sidekiq.org
2020-03-17_19:07:52.07973 rails-background-jobs : 2020-03-17T19:07:52.079Z 6160 TID-oukecdgds INFO: Starting processing, hit Ctrl-C to stop
2020-03-17_19:07:52.10533 rails-background-jobs : 2020-03-17T19:07:52.105Z 6160 TID-oukecdgds INFO: Listening on queues ["adjourned_project_deletion", "admin_emails", "analytics_code_review_metrics", "authorized_keys", "authorized_projects", "auto_devops:auto_devops_disable", "auto_merge:auto_merge_process", "background_migration", "chaos:chaos_cpu_spin", "chaos:chaos_db_spin", "chaos:chaos_kill", "chaos:chaos_leak_mem", "chaos:chaos_sleep", "chat_notification", "container_repository:cleanup_container_repository", "container_repository:delete_container_repository", "create_commit_signature", "create_evidence", "create_github_webhook", "create_note_diff_file", "cronjob:adjourned_group_deletion", "cronjob:adjourned_projects_deletion_cron", "cronjob:admin_email", "cronjob:ci_archive_traces_cron", "cronjob:clear_shared_runners_minutes", "cronjob:container_expiration_policy", "cronjob:elastic_index_bulk_cron", "cronjob:environments_auto_stop_cron", "cronjob:expire_build_artifacts", "cronjob:geo_container_repository_sync_dispatch", "cronjob:geo_file_download_dispatch", "cronjob:geo_metrics_update", "cronjob:geo_migrated_local_files_clean_up", "cronjob:geo_prune_event_log", "cronjob:geo_repository_sync",
... Removed for shortness
]
2020-03-17_19:07:52.16792 rails-background-jobs : 2020-03-17T19:07:52.167Z 6160 TID-oukculo7w INFO: Cleaning working queues
{"timestamp":"2020-03-17T19:09:32.536Z","pid":6087,"message":"PumaWorkerKiller: Consuming 1728.1640625 mb with master and 2 workers."}
2020-03-17_19:09:33.43543 rails-background-jobs : 2020-03-17T19:09:33.435Z 6160 TID-oukculns0 Geo::SidekiqCronConfigWorker JID-2e3b239d2094da7fad6d5bc4 INFO: start
2020-03-17_19:09:33.45063 rails-background-jobs : 2020-03-17T19:09:33.450Z 6160 TID-oukculnlw UpdateAllMirrorsWorker JID-afa9630d7c2d5ed51aaa5c65 INFO: start
2020-03-17_19:09:33.45262 rails-background-jobs : 2020-03-17T19:09:33.452Z 6160 TID-oukcbnrxg ElasticIndexBulkCronWorker JID-fc3bdb8eb995870e3f91e834 INFO: start
2020-03-17_19:09:33.86373 rails-background-jobs : 2020-03-17T19:09:33.863Z 6160 TID-oukculns0 Geo::SidekiqCronConfigWorker JID-2e3b239d2094da7fad6d5bc4 INFO: arguments: []
2020-03-17_19:09:33.90343 rails-background-jobs : 2020-03-17T19:09:33.903Z 6160 TID-oukculns0 Geo::SidekiqCronConfigWorker JID-2e3b239d2094da7fad6d5bc4 INFO: done: 0.468 sec
2020-03-17_19:09:33.94582 rails-background-jobs : 2020-03-17T19:09:33.945Z 6160 TID-oukculnlw UpdateAllMirrorsWorker JID-afa9630d7c2d5ed51aaa5c65 INFO: arguments: []
2020-03-17_19:09:33.96482 rails-background-jobs : 2020-03-17T19:09:33.964Z 6160 TID-oukculnlw UpdateAllMirrorsWorker JID-afa9630d7c2d5ed51aaa5c65 INFO: done: 0.514 sec
2020-03-17_19:09:33.97503 rails-background-jobs : 2020-03-17T19:09:33.974Z 6160 TID-oukcbnrxg ElasticIndexBulkCronWorker JID-fc3bdb8eb995870e3f91e834 INFO: arguments: []
2020-03-17_19:09:33.97913 rails-background-jobs : 2020-03-17T19:09:33.979Z 6160 TID-oukcbnrxg ElasticIndexBulkCronWorker JID-fc3bdb8eb995870e3f91e834 INFO: done: 0.526 sec
2020-03-17_19:10:03.22161 rails-background-jobs : 2020-03-17T19:10:03.221Z 6160 TID-oukculnlw Geo::SidekiqCronConfigWorker JID-869300de0989e93a6b4b5645 INFO: start
2020-03-17_19:10:03.23157 rails-background-jobs : 2020-03-17T19:10:03.231Z 6160 TID-oukcbns4o UpdateAllMirrorsWorker JID-5bc1c21a66e75f65be5acf4c INFO: start
2020-03-17_19:10:03.24389 rails-background-jobs : 2020-03-17T19:10:03.243Z 6160 TID-oukcbnrxg PagesDomainSslRenewalCronWorker JID-9a9d2bf5b0e145c62babd4f8 INFO: start
2020-03-17_19:10:03.24586 rails-background-jobs : 2020-03-17T19:10:03.245Z 6160 TID-oukculns0 ElasticIndexBulkCronWorker JID-102247e89b91d377de7cdb72 INFO: start
2020-03-17_19:10:03.65096 rails-background-jobs : 2020-03-17T19:10:03.650Z 6160 TID-oukculnlw Geo::SidekiqCronConfigWorker JID-869300de0989e93a6b4b5645 INFO: arguments: []
2020-03-17_19:10:03.69169 rails-background-jobs : 2020-03-17T19:10:03.691Z 6160 TID-oukculnlw Geo::SidekiqCronConfigWorker JID-869300de0989e93a6b4b5645 INFO: done: 0.47 sec
2020-03-17_19:10:03.80685 rails-background-jobs : 2020-03-17T19:10:03.806Z 6160 TID-oukcbns4o UpdateAllMirrorsWorker JID-5bc1c21a66e75f65be5acf4c INFO: arguments: []
2020-03-17_19:10:03.83187 rails-background-jobs : 2020-03-17T19:10:03.831Z 6160 TID-oukcbns4o UpdateAllMirrorsWorker JID-5bc1c21a66e75f65be5acf4c INFO: done: 0.6 sec
2020-03-17_19:10:03.89833 rails-background-jobs : 2020-03-17T19:10:03.898Z 6160 TID-oukcbnrxg PagesDomainSslRenewalCronWorker JID-9a9d2bf5b0e145c62babd4f8 INFO: arguments: []
2020-03-17_19:10:04.03984 rails-background-jobs : 2020-03-17T19:10:04.039Z 6160 TID-oukcbnrxg PagesDomainSslRenewalCronWorker JID-9a9d2bf5b0e145c62babd4f8 INFO: done: 0.796 sec
2020-03-17_19:10:04.07467 rails-background-jobs : 2020-03-17T19:10:04.074Z 6160 TID-oukculns0 ElasticIndexBulkCronWorker JID-102247e89b91d377de7cdb72 INFO: arguments: []
2020-03-17_19:10:04.08113 rails-background-jobs : 2020-03-17T19:10:04.080Z 6160 TID-oukculns0 ElasticIndexBulkCronWorker JID-102247e89b91d377de7cdb72 INFO: done: 0.835 sec
2020-03-17_19:11:04.49863 rails-background-jobs : 2020-03-17T19:11:04.498Z 6160 TID-oukdib9vc Geo::SidekiqCronConfigWorker JID-00652032ed8263fe1bada147 INFO: start
2020-03-17_19:11:04.50386 rails-background-jobs : 2020-03-17T19:11:04.503Z 6160 TID-oukdibaeg UpdateAllMirrorsWorker JID-075bc4670ff6900f1aa3cb46 INFO: start
2020-03-17_19:11:04.51826 rails-background-jobs : 2020-03-17T19:11:04.518Z 6160 TID-oukdib7es ElasticIndexBulkCronWorker JID-6a7017573d63428c0c436cfa INFO: start
2020-03-17_19:11:04.81397 rails-background-jobs : 2020-03-17T19:11:04.813Z 6160 TID-oukdib9vc Geo::SidekiqCronConfigWorker JID-00652032ed8263fe1bada147 INFO: arguments: []
2020-03-17_19:11:04.84639 rails-background-jobs : 2020-03-17T19:11:04.846Z 6160 TID-oukdib9vc Geo::SidekiqCronConfigWorker JID-00652032ed8263fe1bada147 INFO: done: 0.348 sec
2020-03-17_19:11:04.95427 rails-background-jobs : 2020-03-17T19:11:04.954Z 6160 TID-oukdibaeg UpdateAllMirrorsWorker JID-075bc4670ff6900f1aa3cb46 INFO: arguments: []
2020-03-17_19:11:04.96655 rails-background-jobs : 2020-03-17T19:11:04.966Z 6160 TID-oukdibaeg UpdateAllMirrorsWorker JID-075bc4670ff6900f1aa3cb46 INFO: done: 0.463 sec
2020-03-17_19:11:04.98213 rails-background-jobs : 2020-03-17T19:11:04.982Z 6160 TID-oukdib7es ElasticIndexBulkCronWorker JID-6a7017573d63428c0c436cfa INFO: arguments: []
2020-03-17_19:11:04.98527 rails-background-jobs : 2020-03-17T19:11:04.985Z 6160 TID-oukdib7es ElasticIndexBulkCronWorker JID-6a7017573d63428c0c436cfa INFO: done: 0.467 sec
2020-03-17_19:12:28.62595 rails-background-jobs : 2020-03-17T19:12:28.625Z 6160 TID-oukcbnufo Geo::SidekiqCronConfigWorker JID-58339cc4a7d2f6a70e5fcf69 INFO: start
2020-03-17_19:12:28.63180 rails-background-jobs : 2020-03-17T19:12:28.631Z 6160 TID-oukcbnumw UpdateAllMirrorsWorker JID-64e394271bf2eab1e6367ad0 INFO: start
2020-03-17_19:12:28.64573 rails-background-jobs : 2020-03-17T19:12:28.645Z 6160 TID-oukcbnspk ElasticIndexBulkCronWorker JID-692aece1c18e33dee62120d4 INFO: start
2020-03-17_19:12:28.98693 rails-background-jobs : 2020-03-17T19:12:28.986Z 6160 TID-oukcbnufo Geo::SidekiqCronConfigWorker JID-58339cc4a7d2f6a70e5fcf69 INFO: arguments: []
2020-03-17_19:12:29.02549 rails-background-jobs : 2020-03-17T19:12:29.025Z 6160 TID-oukcbnufo Geo::SidekiqCronConfigWorker JID-58339cc4a7d2f6a70e5fcf69 INFO: done: 0.4 sec
2020-03-17_19:12:29.07076 rails-background-jobs : 2020-03-17T19:12:29.070Z 6160 TID-oukcbnumw UpdateAllMirrorsWorker JID-64e394271bf2eab1e6367ad0 INFO: arguments: []
2020-03-17_19:12:29.08307 rails-background-jobs : 2020-03-17T19:12:29.083Z 6160 TID-oukcbnumw UpdateAllMirrorsWorker JID-64e394271bf2eab1e6367ad0 INFO: done: 0.451 sec
2020-03-17_19:12:29.20270 rails-background-jobs : 2020-03-17T19:12:29.202Z 6160 TID-oukcbnspk ElasticIndexBulkCronWorker JID-692aece1c18e33dee62120d4 INFO: arguments: []
2020-03-17_19:12:29.20763 rails-background-jobs : 2020-03-17T19:12:29.207Z 6160 TID-oukcbnspk ElasticIndexBulkCronWorker JID-692aece1c18e33dee62120d4 INFO: done: 0.562 secEdited by Oswaldo Ferreira