2024-04-05: RegistryServiceDatabaseApdexSLOViolation

Customer Impact

The problem started at around 8am UTC and got progressively worse, until it suddenly auto resolved at 3pm. This affected our availability on .com which lowered to 99.79% and the availability for container registry was also affected as it went down to 95.56%

Screenshot_2024-04-15_at_09.59.33

Screenshot_2024-04-15_at_10.00.07

source

There were no reports of affected customers, the issue seemed to have occured due to two things happening concurrently:

  • During the incident an unusually large spike of automatic table vacuuming was observed.
  • At the same time GC workers were not resetting properly, which might have caused some kind of soft deadlock between the two until vacuuming completed. This comment by @jdrpereira gives more details on the possible root cause.

Current Status

The incident auto resolved after the automatic table vacuuming was completed. The following issues were created to help prevent this from happening in the future:

📚 References and helpful links

Recent Events (available internally only):

Deployment Guidance

Use the following links to create related issues to this incident if additional work needs to be completed after it is resolved:

Note: In some cases we need to redact information from public view. We only do this in a limited number of documented cases. This might include the summary, timeline or any other bits of information, laid out in our handbook page. Any of this confidential data will be in a linked issue, only visible internally. By default, all information we can share, will be public, in accordance to our transparency value.

Security Note: If anything abnormal is found during the course of your investigation, please do not hesitate to contact security.

Edited by Maina Ng'ang'a