Zoekt pods should have anti affinity rules defined

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

• Close this issue

Background

@igorwwwwwwwwwwwwwwwwwwww has mentioned that 3 out of 4 Zoekt pods are on the same node.

Proposal

We should define node anti affinity rules for Zoekt pods to avoid this situation if we have enough k8s nodes.

We already have affinity value defined https://gitlab.com/gitlab-org/cloud-native/charts/gitlab-zoekt/-/blob/main/templates/stateful_sets.yaml#L169, so we might even not need to update the chart itself and only update the values file in https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com

cc @terrichu