Geo: After DB went down and back up, large number of ProjectImportScheduleWorker jobs ran
Early today the GPRD PostgreSQL node rebooted and PostgreSQL didn't come up automatically (gitlab-com/migration#415 (moved)). After it was manually started, I saw a spike of ProjectImportScheduleWorker
jobs that failed to run due to an UPDATE attempt on the Geo read-only database:
- https://sentry.gitlap.com/gitlab/gitlabcom/issues/195366/
- https://kibana.gprd.gitlab.com/goto/b4ca951dcfab352e756157e1d768f6ba
As far as I can tell, ProjectImportScheduleWorker
is usually instantiated by UpdateAllMirrorsWorker
. It looks like there was only these log messages logged 4 hours beforehand: