Add generic Cells::BulkClaimsWorker for bulk claim operations

Tarun Khandelwalrequested to merge
tk/bulk-claims-worker into master

What does this MR do and why?

Add generic Cells::BulkClaimsWorker for bulk claim operations

Bulk SQL operations (delete_all, upsert_all, insert_all) bypass ActiveRecord callbacks, so the Claimable concern's after_save/ before_destroy hooks never fire -- leaving claims in the Topology Service stale. Making gRPC calls inside DB transactions is also dangerous: a rollback after gRPC success creates orphaned claims, and gRPC on many records can stall the transaction.

This adds a generic Sidekiq worker that runs outside the DB transaction (via run_after_commit) to sync claims for:

  • Route#delete_conflicting_redirects (redirect route deletions)
  • RenameDescendantsService (route renames and redirect inserts)
  • UnconfirmedSecondaryEmailsDeletionCronWorker (email deletions)

Key design decisions:

  • Destroy metadata is built before delete_all since RedirectRoute uses a Proc subject_key that queries the DB for namespace_id
  • Caller-level batching (500 per job) protects Sidekiq/Redis payload limits; BulkClaimService handles gRPC chunking internally
  • Enabled checks (cell config + feature flag) live in the worker to avoid unnecessary DB queries when claims are disabled

References

How to set up and validate locally

Updating a top level group:

  • On the GDK, from the UI, update any top-level group that has sub-groups and/or projects
  • Once done verify the redirect route claim is created for all the sub-groups too:
 gdk psql -d topology_service -c "select bucket_type, bucket_value  from claims;"
 bucket_type |        bucket_value        
-------------+----------------------------
           3 | toolbox
           1 | toolbox-updated
           3 | toolbox/gitlab-smoke-tests
           3 | toolbox/testnewproject
(4 rows)

Deleting and recovering the project:

Recommended to delete all the previous claim so it's easier to see how the claims are getting created

  • Delete a top-level group with group and sub-groups

  • Verify the claims have been created for the redirect route:

 gdk psql -d topology_service -c "select bucket_type, bucket_value  from claims;"
 bucket_type |             bucket_value              
-------------+---------------------------------------
           3 | toolbox-updated
           1 | toolbox-updated-deletion_scheduled-22
           3 | toolbox-updated/gitlab-smoke-tests
           3 | toolbox-updated/testnewproject
(4 rows)
  • now restore the project
  • once restored verify the original redirect route claims are updated and we have claims for the deletion one:
gdk psql -d topology_service -c "select bucket_type, bucket_value  from claims;"
 bucket_type |                       bucket_value                       
-------------+----------------------------------------------------------
           3 | toolbox-updated-deletion_scheduled-22
           1 | toolbox-updated
           3 | toolbox-updated-deletion_scheduled-22/gitlab-smoke-tests
           3 | toolbox-updated-deletion_scheduled-22/testnewproject
(4 rows)

Transfering the group to another one, undo the change and redo it

  • Move a top level group to another group.
  • Verify the claim state after the transfer:
gdk psql -d topology_service -c "select bucket_type, bucket_value  from claims;"

 bucket_type |            bucket_value            
-------------+------------------------------------
           3 | toolbox-updated
           3 | toolbox-updated/gitlab-smoke-tests
           3 | toolbox-updated/testnewproject
(3 rows)
  • Move the group again to be TL and verify the claim state:
gdk psql -d topology_service -c "select bucket_type, bucket_value  from claims;"
 bucket_type |                    bucket_value                     
-------------+-----------------------------------------------------
           3 | flightjs/twitter/toolbox-updated
           1 | toolbox-updated
           3 | flightjs/twitter/toolbox-updated/gitlab-smoke-tests
           3 | flightjs/twitter/toolbox-updated/testnewproject
(4 rows)

notice the new redirect routes and original route getting created.

  • Now, finally do the transfer again to the same group and verify the redirect route claims here
 gdk psql -d topology_service -c "select bucket_type, bucket_value  from claims;"
 bucket_type |                    bucket_value                     
-------------+-----------------------------------------------------
           3 | flightjs/twitter/toolbox-updated
           3 | flightjs/twitter/toolbox-updated/gitlab-smoke-tests
           3 | flightjs/twitter/toolbox-updated/testnewproject
           3 | toolbox-updated
           3 | toolbox-updated/gitlab-smoke-tests
           3 | toolbox-updated/testnewproject
(6 rows)

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Tarun Khandelwal

Merge request reports