## Summary Container registry deletion continues to fail on versions 18.2.x and 18.5.5 for registries with special characters (underscores, dashes) in names, despite related fixes that landed in 18.0.5, 18.1.3, and 18.2.0. Deletion corrupts tag names to empty strings and leaves repositories in unrecoverable `delete_failed` status. ## Relationship to Known Bug For context - the special character deletion bug believed to be fixed is documented in: - [GitLab Support KB Article 21343124384796](https://support.gitlab.com/hc/en-us/articles/21343124384796) - [Container Registry MR #2348 - fix: the Stat call in s3 storage drivers should not rely on lexographical sort only ](https://gitlab.com/gitlab-org/container-registry/-/merge_requests/2348) However, we continue to experience this issue on versions we believe should have been fixed by that aforementioned MR. - **GitLab Version:** 18.5.5-ee (current), 18.2.x (previous incident) - **Deployment:** Self-managed - **Storage Backend:** AWS S3 ## Incidents ### Example Incident 1: November 2024 - **GitLab Version:** 18.2.x - **Registry:** `sgrp_forecast` (contains underscores) - **Result:** Deletion failed, entered `delete_failed` status ### Example Incident 2: (2026-03-03) - **GitLab Version:** 18.5.5-ee - **Registry:** `database-tools-lite` (contains dashes) - **Registry Created:** July 2021 - **Result:** Same symptoms - tag name corrupted to empty string, `delete_failed` status ## Evidence from Current Incident **Database state:** ```sql SELECT id, name, path, status FROM container_repositories WHERE id = 4060; -- id | name | path | status -- -----|------|----------------------------------------------|--------------- -- 4060| "" | redacted/redacted-redacted/.../database-tools-lite | delete_failed ``` **API shows empty tag name:** ```bash curl --header "PRIVATE-TOKEN: <token>" \ "https://gitlab.example.com/api/v4/projects/44435/registry/repositories/4060/tags" # Returns: [ { "name": "", "path": "redacted/redacted-redacted/example-example/database-tools-lite:", "location": "registry.example.com/redacted/redacted-redacted/example-example/database-tools-lite:" } ] ``` **S3 storage shows partial deletion:** ```bash # Tag manifests successfully deleted aws s3 ls s3://bucket/.../database-tools-lite/_manifests/tags/ --recursive # Returns: (empty) # But layers and manifest revisions remain aws s3 ls s3://bucket/.../database-tools-lite/_manifests/revisions/ --recursive # Returns: ~134KB of files aws s3 ls s3://bucket/.../database-tools-lite/_layers/ --recursive # Returns: ~2MB of files ``` ## Problem Description **Pattern observed in both incidents:** 1. Deletion initiated on registry with special characters in name 2. S3 tag manifests successfully deleted 3. Tag name corrupted to empty string in database 4. Repository stuck in `delete_failed` status 5. Retry logic cannot recover (empty name prevents valid API calls) **Based on what is outlined, it's suspected that either:** - The fix from [MR #2348](https://gitlab.com/gitlab-org/container-registry/-/merge_requests/2348) is incomplete - The fix regressed in later versions - A related but different bug exists ## Expected Behavior Registries with special characters in names should be deletable on GitLab 18.2+ (versions containing the fix). ## Actual Behavior Deletion still fails with same symptoms: - Tag name corrupted to empty string - Partial S3 deletion (manifests deleted, layers remain) - Unrecoverable `delete_failed` status - Manual database intervention required ## Impact - **Severity:** High - Affects registries with dashes, underscores in names - Occurs on versions that should have the fix (18.2+, 18.5.5) - Data corruption (original tag name lost) - No automated recovery path - Blocks project moves/transfers ## Workaround Manual cleanup required: ```bash # Delete remaining S3 objects aws s3 rm s3://bucket/docker/registry/v2/repositories/PATH/ --recursive # Clean up database via Rails console container_repository = ContainerRepository.find(4060) container_repository.delete ``` --- The genesis of this work-item is a self-managed ~"GitLab Premium" reaching out to GitLab Support via [ticket](https://gitlab.zendesk.com/agent/tickets/698900), to report this behavior her their example situation used in this opening post.