Restrict ingress-nginx to use the default metallb ippool

Closes #1749 (closed)

Problem with ingres-nginx unit (only unit enabled in workload cluster that deploys a LoadBalancer type svc) was generated by svc.spec.externalIPs (unit values) usage when an additional IPAddressPool.metallb.io was present. This field does not guarantee that MetalLB would provide that IP, based on https://metallb.universe.tf/usage/#requesting-specific-ips. In turn, the field svc.spec.loadBalancerIP would be served by MetalLB, but this usage is discouraged by k8s (called Deprecated):

$ kubectl explain svc.spec.loadBalancerIP
KIND:       Service
VERSION:    v1

FIELD: loadBalancerIP <string>


DESCRIPTION:
    Only applies to Service Type: LoadBalancer. This feature depends on whether
    the underlying cloud-provider supports specifying the loadBalancerIP when a
    load balancer is created. This field will be ignored if the cloud-provider
    does not support the feature. Deprecated: This field was under-specified and
    its meaning varies across implementations. Using it is non-portable and it
    may not support dual-stack. Users are encouraged to use
    implementation-specific annotations when available.


$ kubectl explain svc.spec.externalIPs
KIND:       Service
VERSION:    v1

FIELD: externalIPs <[]string>


DESCRIPTION:
    externalIPs is a list of IP addresses for which nodes in the cluster will
    also accept traffic for this service.  These IPs are not managed by
    Kubernetes.  The user is responsible for ensuring that traffic arrives at a
    node with this IP.  A common example is external load-balancers that are not
    part of the Kubernetes system.

There are 3 units in our stack today that deploy LoadBalancer type Services

• ingress-nginx, used both in management and workload clusters;
• metal3, used only in management cluster;
• k8s-gateway, used only in management cluster;

Since it's theoretically possible for a stack operator to enable creation of multiple IPAddressPool.metallb.io resource in management cluster (using same set of s-u values as used for the workload one in the issue), we're adjusting all 3 of them, also conditionally setting the annotation only for MetalLB usage.

For context: all 3 units are served by the "in-tree" MetalLB (installed inside s-c-c with HelmChart.helm.cattle.io) in RKE2 O/V/M3 infra clusters, but not CAPD ones. For both RKE2 and Kubeadm D infra clusters we use the metallb unit (and a companion capd-metallb-config unit) to have such services exposed and because it wouldn't make sense (network-wise) for metal3 to be deployed inside a CAPD management cluster, we're dropping the conditional (tuple . "capd-metallb-config" | include "unit-enabled"), only having the annotation set for RKE2 O/V/M3 infra clusters.

Last, looks like metallb.universe.tf/loadBalancerIPs svc annotation is not compatible with svc.spec.loadBalancerIP, hence dropping it from values of k8s-gateway for symmetry.