Skip to content

Upgrading to 13.9 makes older Docker images inaccessible

!4962 (merged) bumped container-registry to 3.0.0-gitlab which includes this breaking change:

  • registry/api/v2: Drop support for schema 1 manifests and default to schema 2

Problems

This change instantly made a number of old Docker images inaccessible. I cannot even pull and re-push my images to fix the problem ‼

Release blog posts unclear

There was a confusing deprecation notice in the blog posts leading up to and including 13.9:

  • 13.7 (released 2020-12-22)

    Deprecate pulls that use v1 of the Docker registry API
    GitLab is disabling pulls via the Docker registry v1 APIs on January 22nd, 2021...

  • 13.8 (released 2021-01-22)

    Deprecate pulls that use v1 of the Docker registry API
    GitLab is disabling pulls via the Docker registry v1 APIs on January 22nd, 2021...

  • 13.9

    Deprecate pulls that use v1 of the Docker registry API
    GitLab disabled pulls via the Docker registry v1 APIs on January 22, 2021...

There a multiple problems with these release posts that made it easy to misunderstand the gravity of the change that took place in 13.9:

  1. The heading says that the v1 API is affected. It is not sufficiently clear that all Schema 1 content is affected.
    • I read this originally and thought "no problem, all of our Docker clients are totally new and use the v2 API"
    • It was not clear that content would become inaccessible if not corrected
  2. The text suggests "using a newer Docker client (more recent than 1.12) to rebuild the image and push it to GitLab."
    • What if you can't rebuild the image? How do you keep your data ⁉
  3. I think the author of the GitLab release posts doesn't understand the word "deprecate". "Deprecate" means "to discourage use of"... but the body says "disable".
  4. It is unclear whether "GitLab" means "the GitLab software" or "gitlab.com". The fact that a future date was given implies "gitlab.com" will be affected. Otherwise, why not indicate a future version instead?
  5. The "Deprecation date:" line makes no sense. Again, I think they meant to say "Date at which point this will be disabled"
    • In 13.7 it was the next month
    • In 13.8 it was the same date as the release
    • In 13.9 it was updated to be the current date again

No warning during upgrade

I think it would have been a very good idea to have a pre-flight check in the deb/rpm that checks to see if any of the on-disk images are Schema 1, and prompts the admin to warn that the data will become inaccessible.

No corrective action

There is no indication of how to correct the situation if, like me, you upgraded to 13.9 and then a couple weeks later found that some of your older images are inaccessible.

gitlab#227576 (closed) proposes updating the Web UI to provide more help for users with old images. But it doesn't say how to fix this problem.

IMO This breaking change was not handled very gracefully, and put users with old Docker images in a precarious position.

This was broken in the past (#4145 (closed)) and subsequently fixed (!3061 (merged)).