Fail to push images with layers larger than 10 GB to GitLab container registry backed by S3 storage

Summary

When one or more layers in a container image is over 10 GB in size, pushes to GitLab container registry fails if the backend storage used is S3.

This is a known bug: Not possible to push a layer over 10GB using the S3 driver. and was fixed in upstream Add code to handle pagination of parts. Fixes max layer size of 10GB bug on Registry version 2.8.

This fix was not patched to GitLab container registry: https://gitlab.com/gitlab-org/container-registry.

Reported by a GitLab Ultimate in GitLab customer with 504 seats.

Steps to reproduce

Configure GitLab container registry with S3 as backend storage
Create an image with layers larger than 10 GB in size
Docker push image to the registry.

Example Project

What is the current bug behavior?

Docker pushes for layers larger than 10 GB does not work for registries backed by S3 storage.

What is the expected correct behavior?

Docker pushes for layers larger than 10 GB should work for registries backed by S3 storage.

Relevant logs and/or screenshots

2023-05-15_16:19:08.55633 time="2023-05-15T16:19:08.556Z" level=error msg="manifest unknown" auth_user_name=root auth_user_type=personal_access_token code=MANIFEST_UNKNOWN correlation_id=01H0G3AQJNRQK1KCH0DZ0BXXZ5 detail="unknown tag=187612-main-ef884697" error="manifest unknown: manifest unknown" go_version=go1.18.7 root_repo=redacted vars_name=redacted/redacted vars_reference=187612-main-ef884697 version=v3.63.0-gitlab
2023-05-15_16:36:50.85604 time="2023-05-15T16:36:50.856Z" level=error msg="upload resumed at wrong offset" auth_user_name=root auth_user_type=personal_access_token correlation_id=01H0G4B4XB28MTAB67Q3NHWTE6 go_version=go1.18.7 root_repo=redacted state_offset=10569999840 upload_size=10485760000 vars_name=redacted/redacted vars_uuid=f501169f-0b76-4942-9ccb-aff5aa55d63a version=v3.63.0-gitlab
2023-05-15_16:36:50.87879 time="2023-05-15T16:36:50.878Z" level=error msg="blob upload invalid" auth_user_name=root auth_user_type=personal_access_token code=BLOB_UPLOAD_INVALID correlation_id=01H0G4B4XB28MTAB67Q3NHWTE6 detail="<nil>" error="blob upload invalid: blob upload invalid" go_version=go1.18.7 root_repo=redacted vars_name=redacted/redacted vars_uuid=f501169f-0b76-4942-9ccb-aff5aa55d63a version=v3.63.0-gitlab
2023-05-15_16:46:27.25631 time="2023-05-15T16:46:27.256Z" level=error msg="upload resumed at wrong offset" auth_user_name=root auth_user_type=personal_access_token correlation_id=01H0G4WQSZH19RA8DQVBMXVV61 go_version=go1.18.7 root_repo=redacted state_offset=10569999840 upload_size=10485760000 vars_name=redacted/redacted vars_uuid=78b04244-9472-45eb-96c0-d5cbc0df76f2 version=v3.63.0-gitlab
2023-05-15_16:46:27.28119 time="2023-05-15T16:46:27.281Z" level=error msg="blob upload invalid" auth_user_name=root auth_user_type=personal_access_token code=BLOB_UPLOAD_INVALID correlation_id=01H0G4WQSZH19RA8DQVBMXVV61 detail="<nil>" error="blob upload invalid: blob upload invalid" go_version=go1.18.7 root_repo=redacted vars_name=redacted/redacted vars_uuid=78b04244-9472-45eb-96c0-d5cbc0df76f2 version=v3.63.0-gitlab
2023-05-15_16:56:03.10131 time="2023-05-15T16:56:03.101Z" level=error msg="upload resumed at wrong offset" auth_user_name=root auth_user_type=personal_access_token correlation_id=01H0G5EA5FQG9A05BC21PYRAH2 go_version=go1.18.7 root_repo=redacted state_offset=10569999840 upload_size=10485760000 vars_name=redacted/redacted vars_uuid=28623f2d-403b-4be4-8f35-02d55d044623 version=v3.63.0-gitlab
2023-05-15_16:56:03.12161 time="2023-05-15T16:56:03.121Z" level=error msg="blob upload invalid" auth_user_name=root auth_user_type=personal_access_token code=BLOB_UPLOAD_INVALID correlation_id=01H0G5EA5FQG9A05BC21PYRAH2 detail="<nil>" error="blob upload invalid: blob upload invalid" go_version=go1.18.7 root_repo=redacted vars_name=redacted/redacted vars_uuid=28623f2d-403b-4be4-8f35-02d55d044623 version=v3.63.0-gitlab
2023-05-15_17:05:37.76125 time="2023-05-15T17:05:37.761Z" level=error msg="upload resumed at wrong offset" auth_user_name=root auth_user_type=personal_access_token correlation_id=01H0G5ZVBA6VECK5B4SZ0DCKHN go_version=go1.18.7 root_repo=redacted state_offset=10569999840 upload_size=10485760000 vars_name=redacted/redacted vars_uuid=abafa190-f25a-4d95-b131-839d299f8428 version=v3.63.0-gitlab
2023-05-15_17:05:37.78360 time="2023-05-15T17:05:37.783Z" level=error msg="blob upload invalid" auth_user_name=root auth_user_type=personal_access_token code=BLOB_UPLOAD_INVALID correlation_id=01H0G5ZVBA6VECK5B4SZ0DCKHN detail="<nil>" error="blob upload invalid: blob upload invalid" go_version=go1.18.7 root_repo=redacted vars_name=redacted/redacted vars_uuid=abafa190-f25a-4d95-b131-839d299f8428 version=v3.63.0-gitlab

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info


(For installations with omnibus-gitlab package run and paste the output of:
`sudo gitlab-rake gitlab:env:info`)

(For installations from source run and paste the output of:
`sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)

Results of GitLab application Check

Expand for output related to the GitLab application check

(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true)
(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true)
(we will only investigate if the tests are passing)