Deploy Prometheus to monitor customer apps on Kubernetes
Description
With our support for Prometheus continuing to grow, we should offer the ability to automatically deploy a Prometheus server and configure it to monitor the various environments a project is running in.
Right now, we either ask the customer to bring their own Prometheus server or enable Kubernetes (k8s) monitoring on the bundled Prometheus server. These are not good long term options, however:
- Asking a customer to set up and configure Prometheus is a step that we should not require customers to take. With Kubernetes, spinning up a lightweight app like Prometheus is easy, and we should just take care of it.
- The bundled Prometheus instance should be used to primarily monitor the GitLab service itself. It may not have network reachability to all environments of a project, and best practice for Prometheus is to use multiple servers for different monitoring tasks.
- Dynamic environments like Review apps, in particular, pose a challenge. In these cases, environments will be starting and stopping frequently, and therefore scrape targets will also be changing frequently. There is no way for a customer admin to know and thus define these ahead of time. Similarly attempting to combine both GitLab scrape targets with their permutations plus the complexity of multiple GitLab projects, each having multiple environments, is a significant challenge.
Proposal
For these reasons we should support launching a project-specific Prometheus server automatically if it has k8s integration enabled. We can then use a simple ConfigMap to manage the configuration, and update it on demand. Any time it changes, we can simply send an HTTP request to re-read the configuration. This model has a number of benefits:
- The Prometheus server will be running where the environments are, allowing access to likely private scrape targets.
- Configuration complexity will be reduced, where we only need to worry about the scrape targets for this project alone.
- Fewer scalability challenges.
- Aligning to Prometheus best practices.
For now we can consider requiring direct network access from the GitLab server to the Prometheus server. In the future we should consider leveraging the ability to port forward to Kubernetes pods, to not require any external access in the event GitLab and the Prometheus server are not running in the same network segment.
- On the Kubernetes service success message, if the Prometheus service is inactive, add a link to encourage users to configure it
- Mockup: see mockup
- Text:
You’ve activated the Kubernetes service. To monitor the performance of your environments, [install Prometheus on Kubernetes] or [configure it manually] - “install Prometheus on Kubernetes” starts the installation process and redirects the user to the Prometheus service page
- “configure it manually” simply redirects the user to the Prometheus service page
- On the environment monitoring “Get started” empty state:
- Mockup: see mockup
- Add an “Install Prometheus on Kubernetes” button that starts the “installation” process and redirects the user to the Prometheus service page
- Add a “Manually configure Prometheus” button that simply redirects the user to the Prometheus service page
- On the Prometheus service page:
- Mockup: see the Designs section below
- Move the current manual configuration settings into a “Manual configuration” section
- Add an “Auto configuration” section:
- In this section, have an “Install Prometheus on Kubernetes” button that starts the “installation” process
- Once the “installation” process begins:
- Replace that button with a disabled “Installing Prometheus…” button
- Add an info alert communicating that the Prometheus server is being installed
- Disable the “Manual configuration” section controls
- Once installed, replace the installation button with a “Uninstall Prometheus” button (stops the Prometheus container and cleans up the deployment, replica set, and other k8s settings we created)
- Once the “uninstallation” process begins, replace that button with a disabled “Uninstalling Prometheus…” button and add an info alert communicating that the Prometheus server is being uninstalled
For the configuration itself, since this is in k8s, we can leverage the standard Kubernetes Service Discovery present in Prometheus:
- Add a scrape target for annotated services or pods
- Collect node stats from cAdvisor
- In the future, we can consider allowing custom scrape targets as well
Notes:
- When enabling k8s metrics we should restrict collection to just the namespace for the specific project. (That is configured in the k8s settings)
Designs
- Kubernetes service success message
- Environment monitoring “Get started” empty state
- Prometheus service page:
| Inactive | Installing… | Auto configured | Uninstalling… | Manually configured |
|---|---|---|---|---|
 |
 |
 |
 |
 |
Links / references
Documentation blurb
(Write the start of the documentation of this feature here, include:
- Why should someone use it; what's the underlying problem.
- What is the solution.
- How does someone use this
During implementation, this can then be copied and used as a starter for the documentation.)
-
Developercc @bjk-gitlab
-
-
added ~19492 label
Toggle commit list -
Developer
cc @pedroms for an upcoming 9.2/9.3 item. I would think this could be as simple as a single button to launch it and auto-configure.
There are a few places that we could perhaps put this:
- On the empty state, where if k8s is already configured we can change the button to auto-deploy Prometheus and monitor the apps.
- In the Prometheus service configuration.
- In the Kubernetes configuration, once you were successful in configuring it. I particularly like this one, as we can instantly start getting metrics from k8s as well, so you will start to get data.
-
-
-
-
Developer
@pedroms I'd like to schedule this one as a stretch goal for 9.2, and definitely as deliverable in 9.3. Would you have time to review this soon?
-
-
Developer
@joshlambert I like this idea. Just to be clear, this option would only appear if the project has k8s service enabled, correct?
In the Prometheus service page, if the user has triggered this “Prometheus server auto-deploy and configuration”, does it make sense to offer additional controls (e.g. stopping, restarting)? My question is: since we are doing this automation for the user, what kind of directions should we give them to manage the new Prometheus server? If any…
-
Developer
@pedroms Good questions! Yes, this would require the k8s integration to be enabled. I would think we would launch the prometheus container in the namespace of the project, @bjk-gitlab your thoughts there?
Once launched, we probably should have a button to stop and clean up the prometheus deployment, replica set, etc. Will add to description!
-
-
-
Developer
Yes, launching a Prometheus pod instance within the project namespace is a good way to do it.
The difficulty is how we handle persistence.
I would suggest we use a StatefulSet controller. This might be more complicated, because we will need to integrate with various kuberentes storage schemes like GCE or AWS EBS volumes.
It should be easy enough to detect which type of storage is available to the Kubernetes cluster and auto-provision a volume.
-
Developer
@bjk-gitlab can we just directly use a PVC?
-
-
Developer
Yes, I was talking about using a PVC, but I wasn't sure if it supported automatically selecting a Provisioner.
-
Developer
I don't think we need to worry about that, but I could be wrong. Check out: https://gitlab.com/gitlab-org/kubernetes-gitlab-demo/blob/master/gitlab/gitlab-storage.yml
Nothing in there about specifying a Provisioner.
-
-
Developer
Some notes after talking with @joshlambert about this issue:
- On the Kubernetes service success message, add a link to encourage users to configure the Prometheus service
- On the Prometheus service page, add a button to deploy and configure a Prometheus server
- After deploy, add a button to “stop and destroy” the Prometheus server
@joshlambert feel free to complement with anything else I might have missed. If not, let me know so I can update the description.
-
Developer
Looks good, thanks @pedroms. Only other place we want to add is the empty state, would both enable service and deploy Prometheus server.
-
Developer
@joshlambert yup, thanks for remembering me of that. I've updated the description with this.
-
-
-
-
-
Developer
Moving to 9.3, marking as deliverable.
-
Developer
@joshlambert according to the scope in the description, here are the designs. Question: do we have any idea of how long does the user have to wait for the Prometheus server to be installed on Kubernetes? I'm asking this to assess how the loading state should be designed.
Monitoring empty state
Kubernetes service
Prometheus service
For this page, I had to separate and disable the auto and manual configurations so that they wouldn't conflict with each other if the user decides to switch from one to the other.
Inactive Auto configured Manually configured -
-
Developer
@pedroms I would think a minute or two maximum to spin up the container, unless there is some issue. (Downloading the container image itself, or some resourcing issue with the K8s cluster.)
-
-
added ~974571 and removed ~19492 labels
Toggle commit list -
removed assignee
Toggle commit list -
Developer
@joshlambert ok, thanks for the insight. I've updated the description with the latest designs and flow, adding an installing and uninstalling state. When pressing “Install Prometheus on Kubernetes” on the get started state, the user should be led to the installing state on the Service page. Let me know if you have any concerns.
-
Developer
Thanks @pedroms, looks good! One quick bit of feedback is on the automatically configured state, we should probably just show the namespace and pod name. So for example "gitlab\prometheus".
-
Developer
@joshlambert so instead of “…running on Kubernetes at http://localhost:9000” just show “…running on Kubernetes at gitlab\prometheus”? Is the reference a link to something?
-
-
-
-
-
removed ~897282 label
Toggle commit list -
-
-
removed milestone
Toggle commit list -
-
Developer
Moving to 9.5 due to spill over from 9.3 to 9.4
-
-
-
-
Developer
Based on #35956, we might want to consider moving this button to the k8s service rather than the Prometheus service. See discussion in https://gitlab.slack.com/archives/C433ZLXSB/p1501796117374839.
-
-
-
Developer
@sarrahvesselov @pedroms, as @markpundsack has noted we are looking at a broader Kubernetes configuration method. This would include options to create an actual cluster or link an existing one, and then provision other common components like a CI runner. Prometheus would be part of that, as well.
Based on this new concept, can we revisit the design for this feature? While the broader version will probably not land until a later release, it would be worthwhile to still deliver this in 10.0 to lay the foundation and prove out some of the components. By moving this to the k8s page, we can also better align this.
Does that sound good? I believe you are also copied on the larger issue as well, for input.
-
DeveloperEdited by Joshua Lambert
Noting that we should use the helm chart unless there is a strong reason not to. From a glance, it looks like it should do what we need.
The one concern would be if we deliver this as part of 10.0, ahead of the rest of the items (#35956), how much of the groundwork (Helm Tiller installed, etc.) would we need to pick up.
-
Developer
@sarrahvesselov @pedroms ping back on this issue, to see if we can get some mockups for how this would look if we presented an option to deploy Prometheus on the K8s integration page once configured?
This is the desired end state: #27888, but Prometheus will likely be the leading issue to lay some of the foundation for deploying services into a cluster. Since this is all about cluster management, it makes sense to start with this in the Kubernetes area.
I am thinking it can be as simple as:
- Button to deploy Prometheus once you have a cluster configured, and no other Prometheus server configured. It can be greyed out if you have Prometheus already on, with another server.
- My concern here though is that when you Save an integration, I think you get returned back to the integrations list? That means you wouldn't see this unless you came back which is a problem.
- Once deployed, MVP can just display a temporary banner indicating it is deploying. Later one we can provide more insight into status.
- Button to deploy Prometheus once you have a cluster configured, and no other Prometheus server configured. It can be greyed out if you have Prometheus already on, with another server.
-
-
-
Developer
I removed the
UX Readylabel from this for now. Do you have time to address @joshlambert concerns @pedroms? If not, we can see if @cperessini or another UXer has time to take a peek. -
Developer
Ping back on this one @pedroms, as it's slated for 10.0 so hoping to get some feedback soon. Thanks!
-
-
Developer
@joshlambert I'll pick this issue up. I need to go through the issues and get caught up. I'll have designs next week to review.
-
-
changed the description
Toggle commit list -
-
-
-
Developer
-
-
Developer
@pedroms Mentioned he has time to look at this before he goes on vacation, so he will take back over.
-
Developer
I chatted with @joshlambert about this and how it relates to the “broader Kubernetes (k8s) configuration method”. When we add a button to auto-install/config Prometheus on the k8s page, the Prometheus service page will still need to exist to support manual configs. Because we have a service page dedicated to Prometheus, in the future we'll need to support two routes for auto-installing/config'ing Prometheus: from its dedicated page and from the k8s page (note that this is independent of promoting the k8s service to its own feature page. The dedicated Prometheus service page will always need to:
- allow auto-installing/config'ing Prometheus on k8s cluster(s)
- allow users to input their manual config
- show which configuration method is active (auto on k8s, or manual)
- show the status of the configuration method (installing on k8s…)
As we will always have to support these functions on the Prometheus page, we might as well do them first, in the spirit of the smallest step possible. The installation of services directly from the k8s page is not required to accomplish the intent of this issue so there's no need (at the moment) to add more scope. We will already add a success message on the k8s service that points to the Prometheus auto-config (again, smallest step; see image). The installation of NGINX-ingress, and runners from the k8s page are scheduled for 10.2 so we have time to do this small and include Prometheus as one other installation option in that release. For now, let's limit the scope and move forward with the proposal.
-
Developer
@tauriedavis if you're able, can you please follow this issue as I'll be off? If not, let me know and I'll mention another UX Designer.
🙇 -
Developer
@pedroms This is schedule for 10.1 it looks like. I'll follow along with this issue while youre away but it looks like we may not need to start it for a few weeks anyway? Is that correct?
-
Developer
@tauriedavis yep, that's correct!
-
Developer
@joshlambert Please ping me if you need anything here before 10.1, while Pedro is away!
