Skip to content

Feedback issue: Next generation container registry rollout to self-managed

Next Generation Container Registry

&5521 describes in detail the motivation and technical information about the next-generation container registry. This issue is intended to provide feedback to the GitLab team from self-managed customers who have started or completed the migration to the new registry.

Overview

In Milestone 8.8, GitLab launched the MVC of the Container Registry. This feature integrated the Docker Distribution registry into GitLab so that any GitLab user could have a space to publish and share container images.

However, there was an inherent limitation with Docker Distribution, as all metadata associated with a given image/tag was stored in the object storage backend. This made using that metadata to build API features (like storage usage visibility, sorting, and filtering) unfeasible. The most recent Container Registry update added a new PostgreSQL backend, which is used to store the metadata. Additionally, this new version also includes an automatic online garbage collector to remove untagged images and recover storage space.

Helpful links

Pre Migration Checklist

  • Minimum GitLab version: 17.5
  • Make a backup of the registry's object storage before migrating to the database
  • Ensure that you never run offline garbage collection after migration (check for cron jobs!)
  • Be aware that all untagged images will be removed automatically by online garbage collection
  • Understand once the database is enabled, it must not be disabled again unless you have restored the pre-migration object storage from backup
  • Your GitLab instance does not use Geo — Geo is not yet supported, see Geo: Document Using the New Container Registry ... (#458440 - closed)

Import Time Estimates

These estimates are provided as a rough guide on import time for various size registries. Different environments may produce results that are faster or slower than these time indicates. If you use these figures in your own planning, please consider them as a lower bound and allow for longer import times.

Completed Tests and User Reports

These entries are from imports of test registries performed by GitLab team members, or from reports from users.

Type Size Storage Backend Step One Step Two Step Three Total
Test Registry 100 GiB GCS 23 min 2 min 27 min 52 min
Test Registry 100 GiB S3 6 min 11 sec 1.5 min 8 min
User Registry 160 GiB S3 14 min 1 min 2 Min 17 min
Test Registry 500 GiB GCS 2 hours 9 min 2.5 hours 4.6 hours
Test Registry 500 GiB S3 20 min 90 sec 6 min 28 min
User Registry 14 TiB S3 14.32 hours 19 min 51.5 Hours 66 hours
User Registry (before tag cleanup) 535 TiB NFS 278.4 hours not reported not reported not reported
User Registry (after tag cleanup) 535 TiB NFS 145 hours 1 hour not reported not reported
User Registry 609 TiB S3 344 hours 12.5 hours 615 hours 970 hours

Note: The 535 TiB GCS User Registry's step one and step two completed significantly faster after unused tags were deleted. Removing unused tags, if possible, is recommended before performing imports of large registry deployments.

Instructions - please read before posting!

Thanks for migrating to the next-generation container registry! Before leaving a comment, be sure to checkout our list of things we are already aware of and plan on addressing.

  • This feedback issue is for migrating to the next-generation container registry. If you have other container registry feedback please consider opening a new issue.
  • We will read all of your feedback!
  • We will not respond to all feedback
  • All found issues will be logged in the action summary of this issue

Post-Deployment migrations

Like GitLab Rails, the registry supports both regular schema and post-deployment migrations. For those concerned about downtime during upgrades, starting in GitLab 17.10, you can now skip post-deployment migrations during the upgrade process and apply them manually after the application starts.

This helps prevent unexpected delays during upgrades caused by long-running PDMs, giving you more control over maintenance windows.

Updated documentation:

Actions we are taking from the feedback

  • TBD

Bugs

  • TBD

What we've started looking in to

This is known

  • gitlab-ctl reconfigure removes the registry database configuration on GitLab versions older than 16.4
  • rare image formats will stop the import process on GitLab version older than 16.5

Supported feature status

Feature Description Status Link
Import Tool To migrate existing deployments to the database. Complete Import Tool
Automatic Import Validation Tests that the import maintained data integrity of imported images. Backlog Validate self-managed imports
Geo Support Confirm that Geo replication works with the metadata database. Backlog Geo Support
Migration Runtime Estimates Estimate how long a migration should take in various situations. In progress Test offline migration runtime
HA Database Support HA support for the metadata database. Backlog HA support for other databases
Import Step Retry Retry import steps on failures to increase reliability. Backlog Retry Step on Transient Error
Tag Concurrency Allow simultaneous imports of tags to minimize downtime. Backlog Expose Tag Concurrency Option
DB Provisioning(Omnibus) Automatically provision a new database for Omnibus GitLab installs. Scheduled Provision Database (Omnibus)
Lock Files Use lock files to protect database-managed data. Backlog Lock Files to Preserve Data Consistency
Edited by Hayley Swimelar