Enable session affinity for Auto Deploy with canary
Description
With canary deploys, requests are randomly (or round-robin) distributed to backend pods. This is fine for non-user-facing changes like refactors or performance enhancements, but not so good for user-facing changes. To pick a common example, if you add a new form which posts to a new URL, and a user hits the canary to render the form, but when they submit the form, the request is sent to a non-canary pod, the user may get a 404. Even worse, even for a single page with new CSS or new assets, some parts of the page may fail to render properly.
One solution is to be aware of this constraint and choose not to do canary deploys for user-facing changes. This doesn't scale very well when automated.
Another solution is to use some kind of session affinity so that requests from a single user are sent to the same backend. kube-proxy
supports IP-based affinity, and nginx-ingress
supports cookie-based affinity. Since we use nginx-ingress
in our recommended configurations, we should perhaps change our Auto Deploy helm chart to configure ingress with session affinity.
Note, there are still some drawbacks here. When you add a canary pod, any existing session will continue to route to the pre-existing pods. Only new users will be routed to the canary pod, thus potentially delaying verification of the canary deployment.
Proposal
Links / references
Documentation blurb
Overview
What is it? Why should someone use this feature? What is the underlying (business) problem? How do you use this feature?
Use cases
Who is this for? Provide one or more use cases.
Feature checklist
Make sure these are completed before closing the issue, with a link to the relevant commit.
-
Feature assurance -
Documentation -
Added to features.yml