Convert environment labels to server-side

In order to improve metrics and alert routing and maintenance we should use Prometheus "external labels".

External labels allow Prometheus to assign labels based on its view of the system.

We're currently populating several labels.

  • provider (gcp/azure/etc)
  • region (us-east, etc)
  • monitor (default, app, etc)

We're currently overloading monitor with things like gprd-default.

I propose we change this to add a new env label that contains the environment for the Prometheus server.

This will allow us to transition alert routing to the new labels.