## Context Related to https://gitlab.com/groups/gitlab-org/-/epics/9105+. ## Purpose This epic serves to discuss and collect issues for a data repair strategy needed to fix Container Registry storage usage inconsistencies created by past bugs. The execution of this data repair depends on fixing of all known bugs (https://gitlab.com/groups/gitlab-org/-/epics/9106). ## Problem In https://gitlab.com/groups/gitlab-org/-/epics/9106, and more precisely due to https://gitlab.com/gitlab-org/gitlab/-/issues/217702 and https://gitlab.com/gitlab-org/gitlab/-/issues/379021, we found out that: - Some container repository records were deleted on Rails when the deletion of (at least some of) their tags failed on the registry. - Some project records were deleted on Rails when the deletion of their container repositories failed on the registry - Pending investigation. As a result of the above, there are now non-empty repositories on the registry side that Rails is unaware of, and therefore a storage usage inconsistency between what users see/know and the data remaining in the registry. The investigation/fix of the problems that lead to these inconsistencies belongs to https://gitlab.com/groups/gitlab-org/-/epics/9106+. Here we focus on the data repair that will be required to fix what's behind. ### Examples #### Repository deletion 1. User deletes container repository `a/b/c` on Rails; 1. Rails initiates the deletion of all tags within `a/b/c` _asynchronously_, but this fails; 1. Data is left behind within `a/b/c` on the registry side; 1. Rails deletes the container repository record for `a/b/c` despite the above; 1. User looks at the registry storage usage for project `a/b`; 1. Rails asks the registry for the deduplicated size of all container repositories (that the registry knows about) under `a/b`; 1. Registry returns the size, which includes the data in `a/b/c`; 1. Rails shows the size to users; 1. Users do not understand why usage has not decreased, given they deleted `a/b/c`, and possibly don't even have any other container repositories under that project. #### Project deletion 1. User deletes project `a/b` on Rails; 1. Rails attempts to delete the container repositories under `a/b` _synchronously_, but this fails silently (?); 1. Data is left behind on the registry side; 1. Rails deletes project `a/b`, along with all related container repository records; 1. Users look at the registry storage usage for namespace `a`; 1. Rails asks the registry for the deduplicated size of all container repositories (that the registry knows about) under `a`; 1. Registry returns the size, which includes the data in `a/b*`; 1. Rails shows the size to users; 1. Users do not understand why usage has not decreased, given they deleted project `a`, and possibly don't even have any other projects. ## Plan This should only be executed when https://gitlab.com/groups/gitlab-org/-/epics/9106 is closed and all known bugs resolved. We may find additional problems afterward, so we can repeat the procedure after fixing those new bugs. It's possible that we can merge the repair for both scenarios, but they have been described separately for simplicity. ### Missing repositories under existing projects With a background job, we can: 1. _For each_ project with the container registry feature enabled, where `path` is its full path: 1. Ask the registry for the paginated list of _non-empty_ (at least 1 tag) container repositories under `path`. This can be done by using the _new_ [List Sub Repositories](https://gitlab.com/gitlab-org/container-registry/-/blob/master/docs-gitlab/api.md#list-sub-repositories) API operation; 1. _For each_ returned repository: 1. Safely "find or create" the corresponding container repository record in Rails. The above procedure will re-establish parity between Rails and the registry for _existing_ projects with _missing_ container repositories. Non-empty container repositories that were not successfully (_fully_) deleted will become visible again. _Users_ can then delete them if they wish to. Despite being deleted on the Rails side, users can still pull images from these repositories as long as the associated project exists. Therefore, we should not automatically re-delete these repositories. Deleting data should not happen without explicit user intervention/consent. These steps are essentially what we have been doing (but manually) for incoming support requests (e.g., https://gitlab.com/gitlab-com/support/internal-requests/-/issues/16809). In order to keep track of progress as "we" (background job) progresses, and to allow resuming in case of failure, we must track the progress of the registry data repair for each project (likely using a new column on the `projects` table?) with statuses `in progress`, `failed` or `complete`. Furthermore, to easily identify repositories that have been re-created as part of this data repair, we must also flag those with a special e.g. `re_created_at` field in the `container_repositories` table. #### Caveats - It may be surprising for users to have plenty of repositories coming back to life out of nothing. However, the data was still there, so we're not creating new data. We could find a way to display a warning/label alongside these repositories, but at this point, that's just a nice-to-have. If needed, as long as repositories are correctly flagged as "re-created" as part of this data repair (must), we can always do so in a follow-up later on if needed. - As identified in https://gitlab.com/groups/gitlab-org/-/epics/9107#note_1154773359, by narrowing down the list of projects to those that currently have the container registry feature enabled, we may be missing ghost repositories under projects that had the feature enabled in the past. The alternative is to cycle over all projects, but this poses scale concerns. ### Missing repositories under deleted projects (draft) To tackle missing container repositories under deleted projects, we could do something as above but create the missing projects, so that the creation of missing repositories succeeds. However, I don't think we should re-create the projects and then leave or re-delete them. Projects have many related resources, not just container repositories. Despite possible (unknown) side effects, it would be confusing for users to see deleted projects returning to life. We'd also risk having users creating resources (git, issues, etc.) in those projects in between. Alternatively, to restore parity in missing projects, in a background migration, we could: 1. _For each_ top-level namespace, where `path` is its full path: 1. Ask the registry for the paginated list of _non-empty_ (at least 1 tag) container repositories paths under `path`^<b>*</b>; 1. _For each_ returned repository: 1. Try to match the repository path with any existing project path under the namespace ([code](https://gitlab.com/gitlab-org/gitlab/-/blob/d948af9286721b475214d23416242f1822f7cc19/lib/container_registry/path.rb#L61)); 1. Skip if a matching project is found (no problem); 2. If no matching project is found, send a request to _soft-delete_ the corresponding container repository on the registry side^<b>*</b>. Soft-deleting a repository on the registry side won't cause the deletion of any data. It'll only make the repository and its data invisible to API requests (and we could the same for storage calculations). A soft deletion could be reverted if necessary, but that shouldn't be necessary. If a project is re-created by users and an image is pushed to one of its soft-deleted repositories, the soft-deleted status is automatically reversed thanks to https://gitlab.com/gitlab-org/container-registry/-/merge_requests/884. Any container repository on the registry side that does not match an existing project on the Rails side is effectively inaccessible. The authentication service on Rails always checks that a project exists for a given container repository when receiving an auth request (that precedes any image pull/push). So effectively, it's as if they did not exist at all. Users can't see them or access their data. So why soft deletion vs. actual deletion? 1) for precaution in case the script/background migration has a bug and we soft-delete more than we should 2) the registry doesn't support async deletions, and we know that deleting all repository resources synchronously is a no-go for performance reasons. We can worry about actually deleting these repositories later on. The priority is to make usage accurate _for users_. --- ^<b>*</b> This functionality is not yet supported by the registry API, so we'd need to implement and expose them, restricted to internal/admin use only, with a likely removal once this is fixed. Soft-deleting a repository is already possible and was already done in the past, but only internally. So we'd have to expose it through the API as well.