Enhance Prometheus integration for managed deployments on Kubernetes

Description

We are now able to deploy Prometheus servers to connected Kubernetes clusters with a single button (https://gitlab.com/gitlab-org/gitlab-ce/issues/41053). With GitLab's multi-cluster support, this can mean either a single or multiple Prometheus servers for a given project.

We now need to take this further to make the experience seamless:

1. Automatically enable Prometheus integration, configured for each deployed Prometheus instance
2. Logic to determine which Prometheus server to utilize for a given environment, if multiple clusters are connected.
3. Utilize the Kubernetes API to securely query the Prometheus servers without requiring them to be externally reachable outside the cluster

The general user flow will be:

1. User connects a Kubernetes cluster(s)
2. Deploys Helm Tiller, then Prometheus
3. At this point, Prometheus integration is now automatically enabled. Manually configuring the Prometheus URL is now disabled, instead showing that this is being automatically managed.
4. Monitoring UI elements now appear, and will function as expected for environments which have had a Prometheus server deployed.
5. In the event that the cluster for an environment does not have Prometheus deployed, they will see the empty state.

One note, the currently deployed Ingress does not have metrics enabled or proper annotations for Prometheus monitoring. This is being resolved in https://gitlab.com/gitlab-org/gitlab-ce/issues/39985.

Designs

Note: The general design and user experience for installing applications in the Cluster page is found in https://gitlab.com/gitlab-org/gitlab-ce/issues/38464

View design specs (for spacing, sizes, colors and text copying) — Hide notes in the top-right corner

cc @bjk-gitlab

mentioned in issue omnibus-gitlab#1936 (closed)

added ~19492 label

cc @pedroms for an upcoming 9.2/9.3 item. I would think this could be as simple as a single button to launch it and auto-configure.

There are a few places that we could perhaps put this:

• On the empty state, where if k8s is already configured we can change the button to auto-deploy Prometheus and monitor the apps.
• In the Prometheus service configuration.
• In the Kubernetes configuration, once you were successful in configuring it. I particularly like this one, as we can instantly start getting metrics from k8s as well, so you will start to get data.

added ~122770 label

assigned to @pedroms

mentioned in issue #27521 (closed)

@pedroms I'd like to schedule this one as a stretch goal for 9.2, and definitely as deliverable in 9.3. Would you have time to review this soon?

added Stretch and removed Deliverable labels

@joshlambert I like this idea. Just to be clear, this option would only appear if the project has k8s service enabled, correct?

In the Prometheus service page, if the user has triggered this “Prometheus server auto-deploy and configuration”, does it make sense to offer additional controls (e.g. stopping, restarting)? My question is: since we are doing this automation for the user, what kind of directions should we give them to manage the new Prometheus server? If any…

@pedroms Good questions! Yes, this would require the k8s integration to be enabled. I would think we would launch the prometheus container in the namespace of the project, @bjk-gitlab your thoughts there?

Once launched, we probably should have a button to stop and clean up the prometheus deployment, replica set, etc. Will add to description!

added ~127616 label

added ~901060 label

Yes, launching a Prometheus pod instance within the project namespace is a good way to do it.

The difficulty is how we handle persistence.

I would suggest we use a StatefulSet controller. This might be more complicated, because we will need to integrate with various kuberentes storage schemes like GCE or AWS EBS volumes.

It should be easy enough to detect which type of storage is available to the Kubernetes cluster and auto-provision a volume.

@bjk-gitlab can we just directly use a PVC?

mentioned in issue #30816 (closed)

Yes, I was talking about using a PVC, but I wasn't sure if it supported automatically selecting a Provisioner.

I don't think we need to worry about that, but I could be wrong. Check out: https://gitlab.com/gitlab-org/kubernetes-gitlab-demo/blob/master/gitlab/gitlab-storage.yml

Nothing in there about specifying a Provisioner.

Some notes after talking with @joshlambert about this issue:

• On the Kubernetes service success message, add a link to encourage users to configure the Prometheus service
• On the Prometheus service page, add a button to deploy and configure a Prometheus server
  • After deploy, add a button to “stop and destroy” the Prometheus server

@joshlambert feel free to complement with anything else I might have missed. If not, let me know so I can update the description.

Looks good, thanks @pedroms. Only other place we want to add is the empty state, would both enable service and deploy Prometheus server.

@joshlambert yup, thanks for remembering me of that. I've updated the description with this.

changed milestone to %9.3

added Deliverable and removed Stretch labels

Moving to 9.3, marking as deliverable.

@joshlambert according to the scope in the description, here are the designs. Question: do we have any idea of how long does the user have to wait for the Prometheus server to be installed on Kubernetes? I'm asking this to assess how the loading state should be designed.

Monitoring empty state

Kubernetes service

Prometheus service

For this page, I had to separate and disable the auto and manual configurations so that they wouldn't conflict with each other if the user decides to switch from one to the other.

Inactive	Auto configured	Manually configured

mentioned in commit gitlab-design@04d0e010

@pedroms I would think a minute or two maximum to spin up the container, unless there is some issue. (Downloading the container image itself, or some resourcing issue with the K8s cluster.)

mentioned in commit gitlab-design@8d037d78

removed assignee

@joshlambert ok, thanks for the insight. I've updated the description with the latest designs and flow, adding an installing and uninstalling state. When pressing “Install Prometheus on Kubernetes” on the get started state, the user should be led to the installing state on the Service page. Let me know if you have any concerns.

Thanks @pedroms, looks good! One quick bit of feedback is on the automatically configured state, we should probably just show the namespace and pod name. So for example "gitlab\prometheus".

@joshlambert so instead of “…running on Kubernetes at http://localhost:9000” just show “…running on Kubernetes at gitlab\prometheus”? Is the reference a link to something?

added Stretch and removed Deliverable labels

mentioned in issue #32751 (closed)

mentioned in issue #32638 (moved)

mentioned in merge request !11652 (closed)

mentioned in issue #32831 (closed)

changed milestone to %9.4

mentioned in issue #33556 (moved)

removed milestone

changed milestone to %9.5

Moving to 9.5 due to spill over from 9.3 to 9.4

changed milestone to %10.0

mentioned in issue #35616 (closed)

mentioned in issue #35956 (closed)

Based on https://gitlab.com/gitlab-org/gitlab-ce/issues/35956, we might want to consider moving this button to the k8s service rather than the Prometheus service. See discussion in https://gitlab.slack.com/archives/C433ZLXSB/p1501796117374839.

marked this issue as related to #35956 (closed)

added i2p-on-gke label

@sarrahvesselov @pedroms, as @markpundsack has noted we are looking at a broader Kubernetes configuration method. This would include options to create an actual cluster or link an existing one, and then provision other common components like a CI runner. Prometheus would be part of that, as well.

Based on this new concept, can we revisit the design for this feature? While the broader version will probably not land until a later release, it would be worthwhile to still deliver this in 10.0 to lay the foundation and prove out some of the components. By moving this to the k8s page, we can also better align this.

Does that sound good? I believe you are also copied on the larger issue as well, for input.

Noting that we should use the helm chart unless there is a strong reason not to. From a glance, it looks like it should do what we need.

The one concern would be if we deliver this as part of 10.0, ahead of the rest of the items (https://gitlab.com/gitlab-org/gitlab-ce/issues/35956), how much of the groundwork (Helm Tiller installed, etc.) would we need to pick up.

@sarrahvesselov @pedroms ping back on this issue, to see if we can get some mockups for how this would look if we presented an option to deploy Prometheus on the K8s integration page once configured?

This is the desired end state: https://gitlab.com/gitlab-org/gitlab-ce/issues/27888, but Prometheus will likely be the leading issue to lay some of the foundation for deploying services into a cluster. Since this is all about cluster management, it makes sense to start with this in the Kubernetes area.

I am thinking it can be as simple as:

• Button to deploy Prometheus once you have a cluster configured, and no other Prometheus server configured. It can be greyed out if you have Prometheus already on, with another server.
  • My concern here though is that when you Save an integration, I think you get returned back to the integrations list? That means you wouldn't see this unless you came back which is a problem.
• Once deployed, MVP can just display a temporary banner indicating it is deploying. Later one we can provide more insight into status.

mentioned in issue #27888 (closed)

added UX and removed ~2024186 labels

I removed the UX Ready label from this for now. Do you have time to address @joshlambert concerns @pedroms? If not, we can see if @cperessini or another UXer has time to take a peek.

Ping back on this one @pedroms, as it's slated for 10.0 so hoping to get some feedback soon. Thanks!

assigned to @tauriedavis

@joshlambert I'll pick this issue up. I need to go through the issues and get caught up. I'll have designs next week to review.

changed the description

assigned to @pedroms and unassigned @tauriedavis

changed milestone to %10.1

added Deliverable and removed Stretch labels

With the switch to using Helm, I am moving this to %10.1 and Deliverable. Installation of Helm Tiller (https://gitlab.com/gitlab-org/gitlab-ce/issues/36629) is a pre-requisite.

marked this issue as related to #36629 (closed)

@pedroms Mentioned he has time to look at this before he goes on vacation, so he will take back over.