Chatops Improvment for setting Canary state
Details
- Point of contact for this request: @skarbek
- If a call is needed, what is the proposed date and time of the call: Date and Time
- Additional call details (format, type of call): additional details
SRE Support Needed
We need an improvement made to chatops to slowly enable/disable canary. Right now when the command is initiated, all HAProxies will immediately update all canary backends at nearly the same time. This means that when canary had been drained, thus sitting idle for an extended period of time, the HPA's will have scaled down the count of Pods running. Therefore, when re-enabling canary, all HAProxies will start sending roughly 5% of traffic immediately to the canary backends. This can cause problems as the HPA needs a bit of time to catch up with the immediate spike in load.
Consider improving chatops such that it exposes a new option, may something such as --slowly