A spike of `invalid checksum digest format` errors causes elevated 5xx rate.

Please note: if the incident relates to sensitive data, or is security related consider labeling this issue with security and mark it confidential.

---

Summary

A spike of invalid checksum digest format errors causes elevated 5xx rate.

Service(s) affected : ~"Service:Registry"
Team attribution :
Minutes downtime or degradation :

Timeline

2019-09-18

• 07:02 UTC - A spike of invalid checksum digest format appeared on registry
• 07:05 UTC - On-Call was paged about the incident
• 07:07 UTC - The spike lowered and stays constant in ratio
• 07:10 UTC - The page was acknowledged
• 07:30 UTC - Cause of the page identified and incident issue created
• 08:05 UTC - The automated deploy-pipeline of the customer in question was re-triggered (or triggered by a new commit) by the customer, causing a new image to be built and pushed. The errors went down to 0.
• ...