Plan connectivity between all prometheus instances in all environments, and the alertmanager cluster
Prometheus instances in any environment/VPC/project (gprd, gstg, etc), whether in GCE or GKE, must be able to send alerts to the GKE alertmanager pods. That does literally mean directly to pods, not a service load balancer, because that is how alertmanager is supposed to be used. Prometheus instances send alerts to as many alertmanager instances as they know about, to minimize the probability of an alert not arriving, and the cluster deduplicates the alert.
As a first approximation, this will require IP connectivity between every VPC in all of our Google projects, including to/from the kubernetes pod/service CIDRs (we are using GKE's "alias IP" feature to give VPC-native IPs to pods/services, that can be routed to from GCE). We already have this to an extent - gprd is VPC-peered with gstg and ops for similar reasons.
However, we have strict firewalls rules in place to mitigate the risk of cross-environment talk. We generally don't want services in staging to be able to access production, and vice-versa, by default. We must review all proposals to solve this issue carefully to ensure we don't compromise this effort. For example, allowing all gstg pods to communicate with all gprd pods would not be ok - but allowing those that communication only on an alertmanager port might be ok - it would need to be discussed. As well as GCE firewall, it's worth looking into any GKE/k8s features to see if they can help us here.