Deploy Prometheus to monitor customer apps on Kubernetes
Resources
PM @joshlambert | UX @pedroms
Description
With our support for Prometheus continuing to grow, we should offer the ability to automatically deploy a Prometheus server and configure it to monitor the various environments a project is running in.
Right now, we either ask the customer to bring their own Prometheus server or enable Kubernetes (k8s) monitoring on the bundled Prometheus server. These are not good long term options, however:
- Asking a customer to set up and configure Prometheus is a step that we should not require customers to take. With Kubernetes, spinning up a lightweight app like Prometheus is easy, and we should just take care of it.
- The bundled Prometheus instance should be used to primarily monitor the GitLab service itself. It may not have network reachability to all environments of a project, and best practice for Prometheus is to use multiple servers for different monitoring tasks.
- Dynamic environments like Review apps, in particular, pose a challenge. In these cases, environments will be starting and stopping frequently, and therefore scrape targets will also be changing frequently. There is no way for a customer admin to know and thus define these ahead of time. Similarly attempting to combine both GitLab scrape targets with their permutations plus the complexity of multiple GitLab projects, each having multiple environments, is a significant challenge.
Proposal
For these reasons we should support launching a project-specific Prometheus server automatically if it has k8s integration enabled. We can then use a simple ConfigMap to manage the configuration, and update it on demand. Any time it changes, we can simply send an HTTP request to re-read the configuration. This model has a number of benefits:
- The Prometheus server will be running where the environments are, allowing access to likely private scrape targets.
- Configuration complexity will be reduced, where we only need to worry about the scrape targets for this project alone.
- Fewer scalability challenges.
- Aligning to Prometheus best practices.
For now we can consider requiring direct network access from the GitLab server to the Prometheus server. In the future we should consider leveraging the ability to port forward to Kubernetes pods, to not require any external access in the event GitLab and the Prometheus server are not running in the same network segment.
- On the Kubernetes service success message, if the Prometheus service is inactive, add a link to encourage users to configure it
- Mockup: see mockup
- Text:
You’ve activated the Kubernetes service. To monitor the performance of your environments, [install Prometheus on Kubernetes] or [configure it manually] - “install Prometheus on Kubernetes” starts the installation process and redirects the user to the Prometheus service page
- “configure it manually” simply redirects the user to the Prometheus service page
- On the environment monitoring “Get started” empty state:
- Mockup: see mockup
- Add an “Install Prometheus on Kubernetes” button that starts the “installation” process and redirects the user to the Prometheus service page
- Add a “Manually configure Prometheus” button that simply redirects the user to the Prometheus service page
- On the Prometheus service page:
- Mockup: see the Designs section below
- Move the current manual configuration settings into a “Manual configuration” section
- Add an “Auto configuration” section:
- In this section, have an “Install Prometheus on Kubernetes” button that starts the “installation” process
- Once the “installation” process begins:
- Replace that button with a disabled “Installing Prometheus…” button
- Add an info alert communicating that the Prometheus server is being installed
- Disable the “Manual configuration” section controls
- Once installed, replace the installation button with a “Uninstall Prometheus” button (stops the Prometheus container and cleans up the deployment, replica set, and other k8s settings we created)
- Once the “uninstallation” process begins, replace that button with a disabled “Uninstalling Prometheus…” button and add an info alert communicating that the Prometheus server is being uninstalled
For the configuration itself, since this is in k8s, we can leverage the standard Kubernetes Service Discovery present in Prometheus:
- Add a scrape target for annotated services or pods
- Collect node stats from cAdvisor
- In the future, we can consider allowing custom scrape targets as well
Notes:
- When enabling k8s metrics we should restrict collection to just the namespace for the specific project. (That is configured in the k8s settings)
Designs
- Kubernetes service success message
- Environment monitoring “Get started” empty state
- Prometheus service page:
| Inactive | Installing… | Auto configured | Uninstalling… | Manually configured |
|---|---|---|---|---|
 |
 |
 |
 |
 |
Links / references
Documentation blurb
(Write the start of the documentation of this feature here, include:
- Why should someone use it; what's the underlying problem.
- What is the solution.
- How does someone use this
During implementation, this can then be copied and used as a starter for the documentation.)
-
Developer
cc @bjk-gitlab
-
-
added ~19492 label
Toggle commit list -
Developer
cc @pedroms for an upcoming 9.2/9.3 item. I would think this could be as simple as a single button to launch it and auto-configure.
There are a few places that we could perhaps put this:
- On the empty state, where if k8s is already configured we can change the button to auto-deploy Prometheus and monitor the apps.
- In the Prometheus service configuration.
- In the Kubernetes configuration, once you were successful in configuring it. I particularly like this one, as we can instantly start getting metrics from k8s as well, so you will start to get data.
-
-
-
-
Developer
@pedroms I'd like to schedule this one as a stretch goal for 9.2, and definitely as deliverable in 9.3. Would you have time to review this soon?
-
-
Developer
@joshlambert I like this idea. Just to be clear, this option would only appear if the project has k8s service enabled, correct?
In the Prometheus service page, if the user has triggered this “Prometheus server auto-deploy and configuration”, does it make sense to offer additional controls (e.g. stopping, restarting)? My question is: since we are doing this automation for the user, what kind of directions should we give them to manage the new Prometheus server? If any…
-
Developer
@pedroms Good questions! Yes, this would require the k8s integration to be enabled. I would think we would launch the prometheus container in the namespace of the project, @bjk-gitlab your thoughts there?
Once launched, we probably should have a button to stop and clean up the prometheus deployment, replica set, etc. Will add to description!
-
-
-
DeveloperYes, launching a Prometheus pod instance within the project namespace is a good way to do it.
The difficulty is how we handle persistence.
I would suggest we use a StatefulSet controller. This might be more complicated, because we will need to integrate with various kuberentes storage schemes like GCE or AWS EBS volumes.
It should be easy enough to detect which type of storage is available to the Kubernetes cluster and auto-provision a volume.
-
Developer
@bjk-gitlab can we just directly use a PVC?
-
-
Developer
Yes, I was talking about using a PVC, but I wasn't sure if it supported automatically selecting a Provisioner.
-
Developer
I don't think we need to worry about that, but I could be wrong. Check out: https://gitlab.com/gitlab-org/kubernetes-gitlab-demo/blob/master/gitlab/gitlab-storage.yml
Nothing in there about specifying a Provisioner.
-
-
Developer
Some notes after talking with @joshlambert about this issue:
- On the Kubernetes service success message, add a link to encourage users to configure the Prometheus service
- On the Prometheus service page, add a button to deploy and configure a Prometheus server
- After deploy, add a button to “stop and destroy” the Prometheus server
@joshlambert feel free to complement with anything else I might have missed. If not, let me know so I can update the description.
-
Developer
Looks good, thanks @pedroms. Only other place we want to add is the empty state, would both enable service and deploy Prometheus server.
-
Developer
@joshlambert yup, thanks for remembering me of that. I've updated the description with this.
-
-
-
-
-
Developer
Moving to 9.3, marking as deliverable.
-
Developer
@joshlambert according to the scope in the description, here are the designs. Question: do we have any idea of how long does the user have to wait for the Prometheus server to be installed on Kubernetes? I'm asking this to assess how the loading state should be designed.
Monitoring empty state
Kubernetes service
Prometheus service
For this page, I had to separate and disable the auto and manual configurations so that they wouldn't conflict with each other if the user decides to switch from one to the other.
Inactive Auto configured Manually configured -
-
Developer
@pedroms I would think a minute or two maximum to spin up the container, unless there is some issue. (Downloading the container image itself, or some resourcing issue with the K8s cluster.)
-
-
added ~974571 and removed ~19492 labels
Toggle commit list -
removed assignee
Toggle commit list -
Developer
@joshlambert ok, thanks for the insight. I've updated the description with the latest designs and flow, adding an installing and uninstalling state. When pressing “Install Prometheus on Kubernetes” on the get started state, the user should be led to the installing state on the Service page. Let me know if you have any concerns.
-
Developer
Thanks @pedroms, looks good! One quick bit of feedback is on the automatically configured state, we should probably just show the namespace and pod name. So for example "gitlab\prometheus".
-
Developer
@joshlambert so instead of “…running on Kubernetes at http://localhost:9000” just show “…running on Kubernetes at gitlab\prometheus”? Is the reference a link to something?
-
-
-
-
-
-
-
-
removed milestone
Toggle commit list -
-
Developer
Moving to 9.5 due to spill over from 9.3 to 9.4
