Multi metrics panels don't display correctly on the monitoring dashboard
Summary
As per https://docs.gitlab.com/ee/user/project/integrations/prometheus.html#adding-additional-metrics-premium, it is possible to configure multiple metrics to display on the same chart for the metrics dashboard.
Bug: Instead of multiple metrics displaying, only one of the metrics is shown.
Steps to reproduce
Prerequisites:
- A gitlab-managed cluster connected to a project.
- Prometheus installed on the cluster.
- A deployed environment running on the cluster.
Reproduction:
- Navigate to the prometheus integration settings page (e.g. https://gitlab.com/gitlab-org/monitor/tanuki-inc/-/services/prometheus/edit)
- Add a metric.
- Add another metric, making sure to use the same Title, Group and Y-axis Label as the previous step.
- Navigate to the metrics dashboard (e.g.
https://gitlab.com/gitlab-org/monitor/tanuki-inc/-/environments/1118134/metrics
)
Example Project
https://gitlab.com/gitlab-org/monitor/tanuki-inc/-/services/prometheus/edit
https://gitlab.com/gitlab-org/monitor/tanuki-inc/-/environments/1118134/metrics
What is the current bug behavior?
The metrics panel shown only includes one metric. The missing metrics are also not present on any other panel.
What is the expected correct behavior?
The metrics panel shown should include all metrics configured, overlaid on each other.
Relevant logs and/or screenshots
Config 1 | Config 2 | Config 3 | Result |
---|---|---|---|
Output of checks
(If you are reporting a bug on GitLab.com, write: This bug happens on GitLab.com)
Results of GitLab environment info
Expand for output related to GitLab environment info
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:env:info
)(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production
)
Results of GitLab application Check
Expand for output related to the GitLab application check
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true
)(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true
)(we will only investigate if the tests are passing)
Possible fixes
This seems like a backend issue - the network response for the dashboard is missing the other metrics:
Workarounds
- Add each metric separately (losing the 'multi metric' functionality)
- Create a new custom dashboard
.yml
file and add the multi-metric there (the constraint being that this doesn't work for the default dashboard).