Enable session affinity for Auto Deploy with canary

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Description

With canary deploys, requests are randomly (or round-robin) distributed to backend pods. This is fine for non-user-facing changes like refactors or performance enhancements, but not so good for user-facing changes. To pick a common example, if you add a new form which posts to a new URL, and a user hits the canary to render the form, but when they submit the form, the request is sent to a non-canary pod, the user may get a 404. Even worse, even for a single page with new CSS or new assets, some parts of the page may fail to render properly.

One solution is to be aware of this constraint and choose not to do canary deploys for user-facing changes. This doesn't scale very well when automated.

Another solution is to use some kind of session affinity so that requests from a single user are sent to the same backend. kube-proxy supports IP-based affinity, and nginx-ingress supports cookie-based affinity. Since we use nginx-ingress in our recommended configurations, we should perhaps change our Auto Deploy helm chart to configure ingress with session affinity.

Note, there are still some drawbacks here. When you add a canary pod, any existing session will continue to route to the pre-existing pods. Only new users will be routed to the canary pod, thus potentially delaying verification of the canary deployment.

Proposal

Links / references

Documentation blurb

Overview

What is it? Why should someone use this feature? What is the underlying (business) problem? How do you use this feature?

Use cases

Who is this for? Provide one or more use cases.

Feature checklist

Make sure these are completed before closing the issue, with a link to the relevant commit.

Edited by 🤖 GitLab Bot 🤖