[backport-1.4]: Thanos certificate problem

Issue #2642 (closed) needs to be backported to 1.4.x