Trend analysis and alerts for database queries
https://gitlab.com/gitlab-com/infrastructure/issues/3534#note_54347202 and https://gitlab.com/gitlab-org/gitlab-ce/issues/41988 highlighted a missing piece of monitoring: trend analysis. In particular it would be interesting to monitor the number of updates/dead tuples/etc per table, then send alerts if this increases by say 20%. Using such an alert we would have caught this problem within minutes (depending on the time range of the alert) of the problem happening, not a few days after.
To keep it simple I propose adding an alert that monitors the number of dead tuples. If the amount of dead tuples in the past hour is greater than 24 hours ago (or whatever range we think is better) we send an alert.