Prometheus in GKE does not discover any targets for fluentd

https://prometheus-gke.gprd.gitlab.net/targets

This problem is partly due to incorrect label queries, but from glancing at the fluentd side in minikube it might not be quite that simple.

Update: This issue changed purpose a bit while it was being worked. Initially, it was to fix service discovery for fluentd and calico. While working on it, it was discovered that we were unable to monitor some namespaces, which led to a bit of a rabbit hole that you can read in the comments. This problem, and fluentd scraping, were fixed. https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/10711 was split out to deal with Calico as this issue was getting unwieldy, and Calico was arguably lower priority for metrics.