Disruption for customers pushing to Container Registry using Kaniko

Summary

There seemed to be an issue with pushes to container registry, disproportionately affecting Kaniko user-agents. The errors seem to have dissipated, however the customer who raised the ticket and was affected by this, would like to know if there were any changes on GitLab’s side to explain the disruption. Please see the linked thread for more context and Elasticsearch graphs. https://gitlab.slack.com/archives/CPCJ8CCCX/p1663544987390819

First failed job was Sep 14th 2022 @ 17:09 UTC

First subsequent successful job was Sep 19th 2022 @ 10:17 UTC

The customer was getting specifically errors for DIGEST_INVALID.

Could you please look into why we saw so many 400 errors for PUT from Kaniko user agents during this timeframe? Thanks!

Steps to reproduce

Steps to reproduce the problem, preferably in an ordered list. I was unable to reproduce the issue myself using Kaniko, however there were multiple affected customers.

Ticket Id

Provide a link to the Zendesk ticket if one exists

https://gitlab.zendesk.com/agent/tickets/326705

https://gitlab.zendesk.com/agent/tickets/326488

https://gitlab.zendesk.com/agent/tickets/330753

Architecture

SaaS