Cluster health dashboard data can be of inconsistent time periods
Summary
The lines on the cluster health charts sometimes start and end at different times. This makes the cluster health dashboard look messy (see screenshot). In theory, this should not happen as the queries are supposed to be for the same time periods.
Steps to reproduce
This occurs intermittently, thus may be difficult to reproduce.
- Set up clustering and ensure Prometheus is installed on a project.
- Navigate to to project Kubernetes page: Operations -> Kubernetes
- Observe the cluster health dashboard.
What is the current bug behavior?
The lines on the cluster health dashboard can be of different time periods.
What is the expected correct behavior?
The lines on the cluster health dashboard all cover the same time period (Start and end times line up)
Relevant logs and/or screenshots
Output of checks
This has been reproduced on a local environment, using a minikube setup. There's nothing to suggest (from my limited knowledge) that it won't occur on production.
Possible fixes
In my understanding of the dashboard backend:
- Each separate line is the result of a separate query.
- The queries are cached.
Therefore one might want to look at the cache when trying to find the root cause of this issue.
Note: It's unknown whether this affects other dashboards.