Skip to content

2023-06-13: Sudden spike in gitlab_schema_prevent_write errors at 2023-06-13 16:15 UTC

Customer Impact

Possibly related to a single customer ticket https://gitlab.zendesk.com/agent/tickets/418282 .

Time window was: 2023-06-13 16:12-16:18 UTC

During this time we saw Sidekiq error ratios reach ~10% with 12k jobs failed. This window was short enough that most Sidekiq jobs would have retried successfully and customers may have only noticed some delays in async workloads.

Current Status

@bprescott_ noticed a spike in errors at 2023-06-13 16:15 UTC.

These errors come from our lock write protection which seem to imply someone was trying to write to tables that belong in the main database but they were writing to the ci database.

This appears to have been caused by a misconfiguration in https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/merge_requests/3625 . This resulted in a new PG14 CI cluster coming online and using the same service discovery name as our Main primary database. This resulted in PGBouncer traffic being routed to this new CI database for 6 minutes. Nobody working on this is still around so we don't know why it stopped after 6 minutes but we can see that this Postgres database is stopped.

📚 References and helpful links

Recent Events (available internally only):

Deployment Guidance

Use the following links to create related issues to this incident if additional work needs to be completed after it is resolved:

Note: In some cases we need to redact information from public view. We only do this in a limited number of documented cases. This might include the summary, timeline or any other bits of information, laid out in out handbook page. Any of this confidential data will be in a linked issue, only visible internally. By default, all information we can share, will be public, in accordance to our transparency value.

Edited by Dylan Griffith