Use GitLab to monitor GitLab.com

We should utilize our own observability features ourselves, and the most natural service to monitor is GitLab.com.

This issue is intended to:

• Collect requirements into individual issues, in order to accomplish this and replace our current solution
• Prioritize the issues, so we can begin to use GitLab's Metrics feature for more and more of the daily workflow

We will use an issue for this, instead of an epic, as we want to link issues here but not disrupt the parent epic they are a part of.

Requirements

• Multiple queries for library metrics: https://gitlab.com/gitlab-org/gitlab-ce/issues/54877
  • We lost this support with the migration of metrics into the database, need to re-add.
• Multiple queries for custom metrics: https://gitlab.com/gitlab-org/gitlab-ee/issues/9319
• Support custom time windows: https://gitlab.com/gitlab-org/gitlab-ce/issues/31368
• Multiple dashboards: https://gitlab.com/gitlab-org/gitlab-ee/issues/6019
• Arbitrary zoom in on charts: https://gitlab.com/gitlab-org/gitlab-ce/issues/54878
• Validate we can query Thanos with our Prometheus integration, since we utilize this for GitLab.com scale
• Add Pager Duty and Slack alerts (currently Prometheus directly alerts these targets)