Convert elasticsearch sidekiq nodes to use queue selector
Context: scalability#27 (closed) and scalability#271 (closed)
C3
Production Change - Criticality 3Change Objective | Convert the existing elasticsearch sidekiq node to use the new queue selector |
---|---|
Change Type | ConfigurationChange |
Services Impacted | ServiceSidekiq |
Change Team Members | @cmiskell |
Change Criticality | C3 |
Change Reviewer or tested in staging | @hphilipps |
Dry-run output | N/A |
Due Date | 2020-04-06 01:30UTC (13:30 engineer time) |
Time tracking | 15 minutes |
Detailed steps for the change
-
Merged https://ops.gitlab.net/gitlab-cookbooks/chef-repo/-/merge_requests/3053 -
Run chef-client on sidekiq-elasticsearch-01-sv-gstg -
Check the 'Listen' line in /var/log/gitlab/sidekiq-cluster/current from sidekiq starting up; it should be listening to the 3 queues: cronjob:elastic_index_bulk_cron,elastic_commit_indexer,elastic_indexer
-
Run the apply_to_prod CI pipeline job from the above merge -
Run chef-client on sidekiq-elasticsearch-01-sv-gprd -
Check the 'Listen' line in /var/log/gitlab/sidekiq-cluster/current from sidekiq starting up; it should be listening to the 3 queues: cronjob:elastic_index_bulk_cron,elastic_commit_indexer,elastic_indexer
No additional monitoring required; the 3 jobs noted are the existing jobs already running on those nodes, this is just changing how we get sidekiq to choose to run them, plus an additional prometheus label for the node for the future.
Rollback steps
- Revert the commits in https://ops.gitlab.net/gitlab-cookbooks/chef-repo/-/merge_requests/3053 in a new merge request, and merge.
- Run the apply_to_prod CI pipeline from the new merge request.
- Run chef-client on one or both sidekiq-elasticsearch nodes as necessary.
Changes checklist
-
Detailed steps and rollback steps have been filled prior to commencing work -
Person on-call has been informed prior to change being rolled out
Edited by Craig Miskell