Geo secondaries: Deleted projects may leave repos behind in Gitaly in 16.3 and 16.4
The following discussion from !130565 (merged) should be addressed:
-
@mkozono started a discussion: (+2 comments) question (non-blocking): Does this line fix a bug in project deletion on secondaries in 16.3+? Should we backport it?
Yes, it fixes a bug in project deletion on secondaries in 16.3+. But, the
Geo::Secondary::RegistryConsistencyWorker
and theGeo::DestroyWorker
will tackle the deletion. WDYT?IIRC, deletion via
RegistryConsistencyWorker
can leave behind repos on disk. If true, we can still resolve this thread with a follow up issue and discuss further.
Impact
This can cause the storage on the secondary to become much greater than the primary, equal to the total size of deleted project repos during 16.3/16.4. Therefore this has more impact on large customers.
Workaround
For customers using Gitaly Cluster, I think we could write a procedure to find orphaned project repos in Praefect and delete them. The operation would probably be very manual and take a while (comparison across two PG databases). For those not using Gitaly Cluster, it would take much longer to run, since one side of the comparison has to walk the filesystem.
Potential backport fix
The code change is to add one line project.geo_handle_after_destroy
to ee/app/services/ee/projects/destroy_service.rb
. This looks like less effort than to produce the workaround.