Skip to content

Downsize 1 node of each sidekiq type, take 4

Production Change - Criticality 2 C2

Change Objective Describe the objective of the change
Change Type Type described above
Services Impacted ~"Service:Sidekiq"
Change Team Members @ahmadsherif
Change Severity C2
Buddy check @andrewn
Tested in staging Yes
Schedule of the change 2019-05-23
Duration of the change 15 minutes
Detailed steps for the change. Each step must include: Described below

Terraform plan:

Terraform will perform the following actions:

  ~ module.sidekiq.google_compute_instance.sidekiq_asap[3]
      allow_stopping_for_update: "false" => "true"
      machine_type:              "n1-standard-8" => "custom-4-20480"

  ~ module.sidekiq.google_compute_instance.sidekiq_pages[3]
      allow_stopping_for_update: "" => "true"
      machine_type:              "n1-standard-8" => "n1-standard-4"

  ~ module.sidekiq.google_compute_instance.sidekiq_pullmirror[3]
      allow_stopping_for_update: "" => "true"
      machine_type:              "n1-standard-8" => "n1-standard-4"

  ~ module.sidekiq.google_compute_instance.sidekiq_realtime[3]
      allow_stopping_for_update: "false" => "true"
      machine_type:              "n1-highmem-16" => "n1-standard-8"


Plan: 0 to add, 4 to change, 0 to destroy.

Steps:

  1. Run 
    ssh sidekiq-asap-04-sv-gprd.c.gitlab-production.internal "sudo gitlab-ctl stop sidekiq-cluster"
ssh sidekiq-pages-04-sv-gprd.c.gitlab-production.internal "sudo gitlab-ctl stop sidekiq-cluster"
ssh sidekiq-pullmirror-04-sv-gprd.c.gitlab-production.internal "sudo gitlab-ctl stop sidekiq-cluster"
ssh sidekiq-realtime-04-sv-gprd.c.gitlab-production.internal "sudo gitlab-ctl stop sidekiq-cluster"
  2. Run 
    tf apply -target="module.sidekiq.google_compute_instance.sidekiq_asap[3]" \
-target="module.sidekiq.google_compute_instance.sidekiq_pages[3]" \
-target="module.sidekiq.google_compute_instance.sidekiq_pullmirror[3]" \
-target="module.sidekiq.google_compute_instance.sidekiq_realtime[3]"

Rollback:

  1. Revert https://ops.gitlab.net/gitlab-com/gitlab-com-infrastructure/merge_requests/766
  2. Run steps 1 & 2 from above