Replace Prometheus object definitions with objects from charts

What does this MR do?

Use Helm resources for Prometheus

Replace the Operator specific Prometheus resources with the resources of the Prometheus chart (subchart of the GitLab chart).

Changelog: changed

Migration Notes

• No data must be migrated, because the Prometheus CR was not configured for any persistence.
• The operator will delete the obsolete Prometheus CR and reconcile the Prometheus chart resources. The CR config and the Prometheus chart config are different in many aspects (persistence, alert manager config, scrape config, ...).

Checklist

See Definition of done.

For anything in this list which will not be completed, please provide a reason in the MR discussion.

Required

• Merge Request Title and Description are up to date, accurate, and descriptive
• MR targeting the appropriate branch
• MR has a green pipeline on GitLab.com
• When ready for review, MR is labeled "~workflow::ready for review" per the Distribution MR workflow

Expected (please provide an explanation if not completing)

• Test plan indicating conditions for success has been posted and passes
• Documentation created/updated
• Tests added
• Integration tests added to GitLab QA
• Equivalent MR/issue for omnibus-gitlab opened
• Validate potential values for new configuration settings. Formats such as integer 10, duration 10s, URI scheme://user:passwd@host:port may require quotation or other special handling when rendered in a template and written to a configuration file.

Test Plan

1. Prepare a cluster and install the Prometheus Operator and its CRDs
2. Create a prometheus-k8s ServiceAccount as described in the docs
3. Install Operator (from master or the latest release)
4. Apply a GitLab CR
   1. Verify the CR is reconciled and a Prometheus resource is created
5. Upgrade the Operator to this branch (not just the image, the whole bundle) and enable prometheus in the CR.
   1. Verify the Prometheus CR is deleted
   2. Verify the Prometheus-chart resources are created

Example CR:

apiVersion: apps.gitlab.com/v1beta1
kind: GitLab
metadata:
  name: gitlab
spec:
  chart:
    values:
      prometheus:
        alertmanager:
          enabled: true
        install: true
        nodeExporter:
          enabled: true
        pushgateway:
          enabled: true
    version: 7.3.4

Related issues

Closes #1354 (closed)