Default podAntiAffinity to set pods on different hosts

Idea from gitlab-org/charts/auto-deploy-app!63 (closed). Thanks to @rob_podchaser !

• @rob_podchaser started a discussion: (+1 comment)

  @Negash - This should also support podAntiAffinity (important for High Availability scheduling).

  In fact a good 'convention over configuration' would be to enable podAntiAffinity by default so that replicas are not scheduled on the same node.

  eg. don't schedule pod 'web' on nodes already running 'web':

  metadata:
    name: web
  spec:
    selector:
      matchLabels:
        app: web
    replicas: 3
    template:
      metadata:
        labels:
          app: web
      spec:
        affinity:
          podAntiAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - web
              topologyKey: "kubernetes.io/hostname"

  from

  https://thenewstack.io/implement-node-and-pod-affinity-anti-affinity-in-kubernetes-a-practical-example/