Serialization Error while performing Garbage Collection
While performing the Registry Garbage Collection we are facing the following errors, which are leading to the garbage collection aborting:
failed to garbage collect: enumerating blobs: 5 errors occurred:
* SerializationError: failed to decode REST XML response
status code: 200, request id: JKG..xyz..P0A
caused by: read tcp $IP:$PORT->$IP:$PORT: read: connection reset by peer
or
failed to garbage collect: marking blobs: 1 error occurred:
* s3aws: SerializationError: failed to decode REST XML response
status code: 200, request id: NR..xzy..82Z
caused by: read tcp $IP:$PORT->$IP:$PORT: read: connection reset by peer
Garbage Collection is failing in the mark stage, sweep stage is never reached.
We tried using different registry binary versions but got the the same result everytime:
- 3.45.0
- 3.54.0
- 3.53.0
- 3.39-2
Gitlab is selfhosted in AWS, Registry Backend is AWS S3. We reached out to AWS to confirm, that the connection is not closed by S3 Backend. This is their response:
For each of the unsuccessful requests, I found the same log key value: reason.feClosed = 1. This value means that the front-end closed the connection on S3.
We are running Gitlab 15.1.4 in an Omnibus installation.
Do you have ideas how to troubleshoot this any further?
/CC: @mbruemmer @gyoachum @ckatanda
Salesforce Account Link (Internal Only)
Best regards