Skip to content

Downsize the rest of Sidekiq nodes

Production Change - Criticality 2 C2

Change Objective Describe the objective of the change
Change Type Type described above
Services Impacted ~"Service:Sidekiq"
Change Team Members @ahmadsherif
Change Severity C2
Buddy check @andrewn
Tested in staging Yes
Schedule of the change 2019-05-24
Duration of the change 15 minutes
Detailed steps for the change. Each step must include: Described below

Terraform plan:

Terraform will perform the following actions:

  ~ module.sidekiq.google_compute_instance.sidekiq_asap[4]
      allow_stopping_for_update: "false" => "true"
      machine_type:              "n1-standard-8" => "custom-4-20480"

  ~ module.sidekiq.google_compute_instance.sidekiq_pages[4]
      allow_stopping_for_update: "" => "true"
      machine_type:              "n1-standard-8" => "n1-standard-4"

  ~ module.sidekiq.google_compute_instance.sidekiq_pages[5]
      allow_stopping_for_update: "" => "true"
      machine_type:              "n1-standard-8" => "n1-standard-4"

  ~ module.sidekiq.google_compute_instance.sidekiq_pullmirror[4]
      allow_stopping_for_update: "" => "true"
      machine_type:              "n1-standard-8" => "n1-standard-4"


Plan: 0 to add, 4 to change, 0 to destroy.

Steps:

  1. Run 
    ssh sidekiq-asap-05-sv-gprd.c.gitlab-production.internal "sudo gitlab-ctl stop sidekiq-cluster"
ssh sidekiq-pages-05-sv-gprd.c.gitlab-production.internal "sudo gitlab-ctl stop sidekiq-cluster"
ssh sidekiq-pages-06-sv-gprd.c.gitlab-production.internal "sudo gitlab-ctl stop sidekiq-cluster"
ssh sidekiq-pullmirror-05-sv-gprd.c.gitlab-production.internal "sudo gitlab-ctl stop sidekiq-cluster"
  2. Run 
    tf apply -target="module.sidekiq.google_compute_instance.sidekiq_asap[4]" \
-target="module.sidekiq.google_compute_instance.sidekiq_pages[4]" \
-target="module.sidekiq.google_compute_instance.sidekiq_pages[5]" \   
-target="module.sidekiq.google_compute_instance.sidekiq_pullmirror[4]"

Rollback:

  1. Revert https://ops.gitlab.net/gitlab-com/gitlab-com-infrastructure/merge_requests/766
  2. Run steps 1 & 2 from above