[PROD] Scale out web fleet
Production Change - Criticality 2 ~"2"
Change Objective | Scale out web fleet in prod |
---|---|
Change Type | Scaling |
Services Impacted | web |
Change Team Members | @aamarsanaa |
Change Severity | S3 |
Buddy check or tested in staging | TBD |
Schedule of the change | June 13 23:00PM UTC+8 |
Duration of the change | 30 mins |
Detailed steps for the change. Each step must include: | - pre-conditions for execution of the step, - execution commands for the step, - post-execution validation for the step , - rollback of the step |
Pre-conditions
- 28 web nodes + 2 web-cny nodes
Execution
-
Terraform change to increase web
count from 28 to 36. (36 so that it is nearly 30%) -
Chef change to add the new nodes to gprd-base-lb-fe
role
Post-execution validation
-
Make sure nodes are provisioned successfully in GCP -
Make sure nodes are registered by the fe LBs and traffic is being sent to them -
Make sure new nodes successfully are processing requests
Edited by Amarbayar Amarsanaa