2020-02-11: High insert rate for services table causing higher web latencies

Summary

We see high insert rates for the services db table causing higher web latencies.

(https://dashboards.gitlab.net/d/RZmbBr7mk/gitlab-triage?orgId=1&fullscreen&panelId=8&from=1581399974013&to=1581413232942)

(https://dashboards.gitlab.net/d/general-service/general-service-platform-metrics?orgId=1&fullscreen&panelId=10&from=1581400421276&to=1581413397765)

More information will be added as we investigate the issue.

Timeline

All times UTC.

2020-02-11

• 05:59 - First service template created
• 06:31 - High insert rate into services db table starts
• 07:16 - db replication alert fired
• 08:18 - pgbouncer reaches saturation, slowing down the whole site
• 08:37 - Draining canary to see if a change could be the cause (no effect).
• 09:04 - service template creation set to false, insert rate dropping down to normal level; pgbouncer, web saturation going down
• 09:19 - enabling canary again
• 09:48 - draining canary again