Test Prometheus 3.0

Since Prometheus 3.0 has released, and a subsequent patch release 3.1 to fixing some bugs, we need to test it for stability and compatibility with Mimir.

• 3.0 Release notes
• Announcement Blog

Going over the release notes there is a few important changes we should just validate:

• Remote Write 2.0
• [CHANGE] SD: Enable the new service discovery manager by default. This SD manager does not restart unchanged discoveries upon reloading. This makes reloads faster and reduces pressure on service discoveries' sources. The corresponding new-service-discovery-manager feature flag has been removed.
  • This particular change should be a good improvement for us, and while it shouldn't case any issues, I'm noting it here to be sure we at least are aware of this change in default behaviour.
• [CHANGE] Agent mode has been promoted to stable. The feature flag agent has been removed. To run Prometheus in Agent mode, use the new --agent cmdline arg instead
• [CHANGE] Logging: the format of log lines has changed a little, along with the adoption of Go's Structured Logging package.
  • Not a concern, but we need to ensure logs in Elastic Cloud are still readable and parsed correctly. Might need to update the mapping.
• [CHANGE] UTF-8: Enable UTF-8 support by default. Prometheus now allows all UTF-8 characters in metric and label names. The corresponding utf8-name feature flag has been removed.
  • I don't expect any issues here but it could alter some labels or start collecting labels that might have been previously dropped. Probability of any of this is very low.
• [CHANGE] Set the GOMAXPROCS variable automatically to match the Linux CPU quota. Use --no-auto-gomaxprocs to disable it. The auto-gomaxprocs feature flag was removed
  • We currently have this set, and will need to remove it.

There is some additional relevant info in the Prometheus 3.0 Migration Guide.