Skip to content

Remove ETag checks for multipart uploads to object storage

Stan Hu requested to merge sh-loosen-etag-comparison into master

Minio recently updated its behavior to generate an ETag that is NOT an MD5 sum of the file (https://github.com/minio/minio/pull/7609), which causes all uploads to break with GitLab. Other S3 providers, such as Exoscale Object Storage, also behave the same way.

According to https://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadComplete.html:

The entity tag is an opaque string. The entity tag may or may not be an MD5 digest of the object data. If the entity tag is not an MD5 digest of the object data, it will contain one or more nonhexadecimal characters and/or will consist of less than 32 or more than 32 hexadecimal digits.

To fix this issue, we now disable the MD5/ETag checking for multipart uploads but keep it for the PUT object API. Since Google Cloud Storage currently only uses the PUT object API, we retain some integrity checks there.

Edited by Stan Hu

Merge request reports