Skip to content

fix: Use instance as join label for redis-ops rates

Before we were using pod and fqdn as the join labels. This would mean that when a pod had the same name across environments, we'd select series for both environments and display them on the graph.

Changing that to use the instance label, which uses an ip-adress:port in the k8s case solves this.

Dashboards at a prolematic time:

pre gstg
Before image https://dashboards.gitlab.net/d/redis-ratelimiting-main/redis-ratelimiting-overview?orgId=1&var-PROMETHEUS_DS=Global&var-environment=pre&from=1665875796404&to=1666097007299 image https://dashboards.gitlab.net/d/redis-ratelimiting-main/redis-ratelimiting-overview?orgId=1&var-PROMETHEUS_DS=Global&var-environment=gstg&from=1665875796404&to=1666097007299
After image https://dashboards.gitlab.net/dashboard/snapshot/53kjF64frfjA9uOY5Ro82GzI55gEGjeW?orgId=1&var-environment=pre&from=1665875796404&to=1666097007299 image https://dashboards.gitlab.net/dashboard/snapshot/53kjF64frfjA9uOY5Ro82GzI55gEGjeW?orgId=1&var-environment=gstg&from=1665875796404&to=1666097007299

This shows that redis-ratelimiting-node-2 is supposed to be the primary for pre. While redis-ratelimiting-node-0 is the primary for gstg at the beginning of that range. But in the before graph, both nodes show up cross-environment, because they have the same pod label.

At the end of the graph there, the pre instance switches ip, while the primary for gstg switches from node-0 to node-2.

For gitlab-com/gl-infra/scalability#1875 (comment 1140950779)

Edited by Bob Van Landuyt

Merge request reports