Deploy Kubernetes web components in production main/canary stages

This is to track all the steps needed to get canary web completely running in Kubernetes

Todo:

• Add a web node pool to Kubernetes cluster gprd-gitlab-gke via terraform
• Add the cny web deployment to k8s-workloads/gitlab-com and get the pods running with no traffic
• Determine how to make haproxy able to do separate health checks depending on backend. Our k8s backends will need to check /-/k8s/web/-/readiness while our vms check just /-/readiness
• HAProxy cookbook updated with better canary configuration
• Add web cny in k8s to haproxy
• Checkpoint and confirm configuration audit is complete and configuration is in sync #1065 (closed)
• Checkpoint validate all mitigations and migration blockers are in a safe state
• Slowly transfer cny traffic from vms, to mixed, to k8s
• Move traffic on main stage from vms, to k8s
• Finalize chef configuration