Migrate Free tier images with more than 100 tags

Context

This is part of the work to upgrade and migrate the GitLab.com container registry to a new version backed by a metadata database and online garbage collection (&5523 (closed)). This will be achieved following the gradual migration plan detailed in container-registry#374 (closed).

Task

Perform the Phase 2 (existing container repositories) general percentage-based rollout for Free tier namespace images with > 100 tags.

Monitoring and Known Issues

These are the known issues to date and the json.error message pattern that will appear in the logs for each:

  • Migration failures due to list/indexes with bro... (container-registry#651 - closed) - Rare

    pre importing tagged manifests: pre importing manifest: pre importing manifest list: retrieving referenced manifest "<digest>" from filesystem: unknown manifest name=<repo> revision=<digest>
  • Pre-import failures due to missing manifest rev... (container-registry#654 - closed) - This is the most prominent issue

    pre importing tagged manifests: pre importing manifest: retrieving manifest "<digest>" from filesystem: unknown manifest name=<repo> revision=<digest>
  • Cloud Storage hiccups. Nothing to do, the import is retried automatically - Very rare

    pre importing tagged manifests: pre importing manifest: pre importing manifest: transferring blob with digest sha256:XXX: failed to transfer blob digest="sha256:XXX" cleanup=true cleanupError="gcs: context canceled": "partial transfer from <gcs_path_1> to <gcs_path_2>: copying data from source to destination: context canceled"
  • Pre-imports failing due to failure obtaining co... (container-registry#660 - closed) - Rare

    pre importing tagged manifests: pre importing manifest: pre importing manifest: obtaining configuration payload: invalid checksum digest format
  • Importer: Gracefully handle empty layer link co... (container-registry#658 - closed) - Rare

    pre importing tagged manifests: pre importing manifest: pre importing manifest: importing layers: checking for access to blob with digest <digest> on repository <name> invalid checksum digest format
  • Imports failing with invalid memory address or ... (container-registry#662 - closed) - Very rare

  • pre importing tagged manifests: pre importing manifest: pre importing manifest: importing layers: transferring blob with digest <digest>: %!v(PANIC=Error method: runtime error: invalid memory address or nil pointer dereference)

If you're monitoring the migration, use this log view to identify import failures.

If you notice any imports failing with a json.error that does not match one of the above, please disable the main feature flag (container_registry_migration_phase2_enabled) immediately and report the error here for investigation.

Edited May 13, 2022 by Tim Rizzi
Assignee Loading
Time tracking Loading