Prometheus server fails to get metrics from deployed environments
Metrics dashboard is showing the message
No data found
Steps to reproduce
Metrics section inside of
What is the current bug behavior?
There are no metrics shown.
The only way of getting it to work is to completely reinstall GDK.
These are the things that have been tried without success:
- Reconnecting an existing cluster with a project
- Recreating the project and connecting it to the existing cluster
- Recreating both the cluster and the project and doing everything from scratch
- This message appeared in the logs for
unable to fully collect metrics: unable to fully scrape metrics from source
/additional_metrics.jsonreturns a 200 response, but data is empty. e.g:
- All responses are empty when hitting the Prometheus proxy API directly
More information if you want to dig deeper into this issue:
- In the Rails console, run
Deployment.find(<id>).additional_metricsand work the way inside the method. It may help narrow down
- Said on Slack about the issue:
"The most likely theory I think is that the reactive cache is still waiting for results so GitLab is returning nothing."
If you encounter this issue, please contact @tkuah