Thanos frontend unable to parse host
Problem
Inside of ops-gitlab-gke
we have the following pods running:
steve@bastion-01-inf-ops.c.gitlab-ops.internal:~$ kubectl -n monitoring get po | grep 'thanos-query-frontend'
thanos-query-frontend-7dbdb8b78b-5txlk 1/1 Running 3 34d
thanos-query-frontend-7dbdb8b78b-nwwls 1/1 Running 1 34d
thanos-query-frontend-7dbdb8b78b-txng7 1/1 Running 4 34d
Now if we look at the error logs we see the following errors:
{"caller":"http.go:48","err":"address thanos-query.ops.gitlab.net: missing port in address","http.component":"server","level":"error","msg":"failed to parse host port for http log decision","protocol":"http","ts":"2021-10-18T07:32:18.144794765Z"}
{"caller":"http.go:48","err":"address thanos-query.ops.gitlab.net: missing port in address","http.component":"server","level":"error","msg":"failed to parse host port for http log decision","protocol":"http","ts":"2021-10-18T07:35:37.812742914Z"}
Where is coming from
This is coming from HTTPMiddleware
What are we missing from failing to
Depending on the decider it can log information such as start and finish time of a request which might be useful.
Hypothesis
- It might be the GKE service is stripping the port
- Thanos is somehow getting the wrong host