Geo Container Registry replication missing manifest revision links, causing accumulating file deficit

Summary

Geo Container Registry replication is missing manifest revision link files, causing an accumulating deficit of files on secondary S3 storage with each image update. The missing files are consistently _manifests/revisions/sha256/<digest>/link files.

Steps to reproduce

  1. Set up Geo with Container Registry using S3 storage
  2. Push a new image to primary: docker push primary.example.com/root/project:latest
  3. Compare S3 file counts: aws s3 ls s3://primary-registry --recursive | wc -l vs aws s3 ls s3://secondary-registry --recursive | wc -l
  4. Update the image (e.g., add EXPOSE 8080 to Dockerfile) and push again: docker push primary.example.com/root/project:latest
  5. Compare S3 file counts again
  6. Repeat steps 4-5 multiple times

Expected: File counts should match after each sync Actual: Secondary gets one fewer file with each update, and the deficit accumulates

What is the current bug behavior?

When updating an image and pushing to the same tag, the secondary registry is consistently missing one _manifests/revisions/sha256/<digest>/link file per update. Example pattern after 3 updates:

Initial push:  50 files on primary, 50 on secondary (0 missing)
First update:  57 files on primary, 56 on secondary (1 missing)
Second update: 65 files on primary, 63 on secondary (2 missing) 
Third update:  79 files on primary, 76 on secondary (3 missing)

Missing files are always under _manifests/revisions/ and are link files:

docker/registry/v2/repositories/root/project/_manifests/revisions/sha256/ad08a9456f7bbf4562588f7d60e4331dd7639a163978c2635bb28bd73f26b8d4/link
docker/registry/v2/repositories/root/project/_manifests/revisions/sha256/bc8e750e8203ce9d1a484aa26c2cfe871f9e2977063e4cc4e8a63baa28c418de/link
docker/registry/v2/repositories/root/project/_manifests/revisions/sha256/f77c8368469c36eecf28b18f610db9c50f139a7d599077f30211b91457cda123/link

What is the expected correct behavior?

All files, including manifest revision links, should be replicated to the secondary registry. The file count should match between primary and secondary after each sync completes.

Understanding Docker Registry v2 Storage

The Docker Registry v2 stores manifests in two parallel structures:

  1. Tags (mutable): _manifests/tags/<tag-name>/current/link

    • Points to the current manifest for a tag (e.g., latest)
    • Updated when you push to the same tag

  2. Revisions (immutable, content-addressable): _manifests/revisions/sha256/<digest>/link

    • Points to manifests by their content digest (SHA256)
    • Should exist for EVERY manifest version that was ever pushed
    • Enables access by digest: docker pull registry.com/project@sha256:abc123...
    • Critical for garbage collection safety and registry consistency

Root Cause Analysis

In ee/app/services/geo/container_repository_sync.rb, the sync_tag method pushes manifests in two ways:

  1. Submanifests (in multi-arch images): Pushed by digest → Creates revision link ✅
  2. Main tag-level manifest: Pushed by tag name only → Creates tag link but NO revision link ❌
def sync_tag(tag)
  manifest = client.repository_raw_manifest(repository_path, tag[:name])
  manifest_parsed = Gitlab::Json.safe_parse(manifest)

  if LIST_MANIFESTS.include? manifest_parsed['mediaType']
    # ... handle submanifests ...
    manifest_parsed['manifests'].each do |submanifest_ref|
      # This creates revision links ✅
      container_repository.push_manifest(
        submanifest_ref['digest'],  # ← Pushed by digest
        submanifest_raw,
        submanifest_parsed['mediaType']
      )
    end
  else
    sync_manifest_blobs(manifest_parsed)
  end

  # Only pushed by tag name, NOT by digest ❌
  manifest_media_type = manifest_parsed['mediaType'] || ContainerRegistry::Client::OCI_MANIFEST_V1_TYPE
  container_repository.push_manifest(tag[:name], manifest, manifest_media_type)
end

What happens on each sync:

  • We fetch the current manifest for the tag from primary
  • We push this manifest by tag name only (line 74)
  • We NEVER push it by its digest
  • This means the revision link for the current manifest is never created
  • This happens for EVERY manifest version that gets assigned to a tag

Example sequence:

  1. Initial push of latest with manifest sha256:aaaa...:

    • Secondary creates: _manifests/tags/latest/current/link ✅
    • Secondary missing: _manifests/revisions/sha256/aaaa.../link ❌

  2. Update latest to new manifest sha256:bbbb...:

    • Secondary updates: _manifests/tags/latest/current/link → sha256:bbbb ✅
    • Secondary missing: _manifests/revisions/sha256/aaaa.../link ❌ (never created)
    • Secondary missing: _manifests/revisions/sha256/bbbb.../link ❌ (not created now either)

  3. Each subsequent update adds one more missing revision link

Impact

  • Data Integrity: Incomplete replication of registry metadata
  • Content-addressable access broken: docker pull registry.com/project@sha256:abc... fails on secondary
  • Garbage collection risk: Missing revision links could lead to manifests being considered orphaned
  • Disaster recovery compromised: Secondary registry is not a complete replica
  • Accumulating deficit: Problem gets worse over time with each image update
  • Registry consistency failures: Registry expects both tag and revision structures to be in sync

Workaround

There is no automated workaround. Manual remediation options:

  1. Manual S3 file copy: Identify and copy missing _manifests/revisions/sha256/<digest>/link files from primary to secondary S3 bucket

  2. Custom remediation script: Walk the primary registry storage directory structure, enumerate all revision links under _manifests/revisions/, and copy missing ones to secondary

Why resync won't work:

  • The sync logic only processes tags that differ between primary and secondary (see tags_to_sync = primary_tags - secondary_tags)
  • Revision links are not discoverable through the registry tag list API
  • Old manifest versions that were previously assigned to tags cannot be enumerated via the API

Note: Once the code fix is deployed, existing missing revision links will remain missing unless a backfill migration is created to reconstruct them from primary storage.

Possible fixes

The fix should ensure that when pushing a manifest by tag, we also push it by digest to create the revision link:

# Push by tag name
container_repository.push_manifest(tag[:name], manifest, manifest_media_type)

# Also push by digest to create revision link
manifest_digest = Digest::SHA256.hexdigest(manifest)
manifest_digest_with_prefix = "sha256:#{manifest_digest}"
container_repository.push_manifest(manifest_digest_with_prefix, manifest, manifest_media_type)

Alternatively, investigate if the Docker Registry v2 API automatically creates revision links when pushing by tag (in which case this might be a registry version/configuration issue).

For existing deployments: A backfill script/migration will be needed to:

  1. Enumerate all manifests in primary registry storage
  2. Compare with secondary registry storage
  3. Create missing revision links on secondary

Affected Code

  • ee/app/services/geo/container_repository_sync.rb:45-75 (sync_tag method)
  • ee/lib/ee/container_registry/client.rb:20-26 (push_manifest method)
Edited by 🤖 GitLab Bot 🤖