pgbouncer exporter is incorrect
Slack thread: https://gitlab.slack.com/archives/CB7P5CJS1/p1566408016230500
There appears to be a fair amount of evidence pointing to pgbouncer_exporter
providing the wrong data at present:
From @NikolayS's description:
To me, it looks like some kind of problem with exporter -- I don't see imbalance, I see that for some nodes zeroes are reported... The 1st graph, for example, which is queries per second, I suppose. A lot of zeroes and couple of lines which are very close to each other. Nodes 07 and 08 are at levels 12-13K QPS (until the moment 07 became the new master), node 06 shows 0 01 and 09 are not used, as well as former master 04. 02, 03, 05 are missing completely... Manual check: production$1887857 Hard to observe, but it's possible. For node 06, for example, query count was 5152713234 for pool gitlabhq_production at Aug 21 10:50:56. And then it's 5152964709 at Aug 21 10:51:17. Increased by 251475 during ~21 seconds, it gives us QPS ~11975. Matches with node 08 which is presented on the graph. But why others are not...
We are currently experience a high number of incidents due to pgbouncer. pgbouncer_exporter
is one of our primary views into this subsystem and many of the conclusions we have drawn are based on its.
If it is not working correctly we should address this immediately.
Interestingly, Gocardless (cc former employee @craigf) forked @stanhu's fork and have made substantial changes. We should review these to understand whether it would be worth incorporating any of them: https://github.com/stanhu/pgbouncer_exporter/compare/master...gocardless:master