DNS changes to performance.gitlab.net broke existing metrics
It looks like a new server performance-tr was created for InfluxDB. The DNS name for performance.gitlab.net got mapped to this new server.
This broke our internal metrics dashboard because while the GitLab.com was sending to an internal IP (10.x.x.x), Grafana data sources were mapped to performance.gitlab.net.
This new performance.gitlab.net did not have users or the gitlab database configured for it. To do this, we ran:
$ influx
CREATE USER root WITH PASSWORD '<password from 1password> WITH ALL PRIVILEGES;
CREATE DATABASE gitlab;
CREATE USER gitlab WITH PASSWORD '<password from 1password>`;
GRANT ALL ON gitlab to gitlab;
https://performance.gitlab.net/datasources (requires org admin access) contains all the data sources. We temporarily fixed the issue by changing Grafana to use the internal IP instead of the hostname.
/cc: @ernstvn, @tmaczukin, @maratkalibek
Edited by Stan Hu