Nested Groups and Project will not delete from the top level

Summary

Nested Groups and Project will not delete from the top level

Steps to reproduce

Create a Group, and within that group create many (in this case 250) sub groups. From within those sub groups create several projects. When deleting a group, that contains groups, that contains projects the sidekiq job fails. If you try and delete a sub group with projects that is successful.

Example Project

https://gitlab.com/gitlab-org/quality/performance-data/-/tree/master can be used to create the project

What is the current bug behavior?

image

What is the expected correct behavior?

The group should be removed

Relevant logs and/or screenshots

Manually deleting form the rails console GitLab: 13.5.1-ee (a4cc9d13) EE

GitLab Shell: 13.11.0

PostgreSQL: 11.6,

Loading production environment (Rails 6.0.3.3)

irb(main):001:0> GroupDestroyWorker.new.perform(11, 1)

Scoped order is ignored, it's forced to be batch order.

Scoped order is ignored, it's forced to be batch order.

Scoped order is ignored, it's forced to be batch order.

Scoped order is ignored, it's forced to be batch order.

Scoped order is ignored, it's forced to be batch order.

Scoped order is ignored, it's forced to be batch order.

Scoped order is ignored, it's forced to be batch order.

Scoped order is ignored, it's forced to be batch order.

Scoped order is ignored, it's forced to be batch order.

Scoped order is ignored, it's forced to be batch order.

Traceback (most recent call last):

   16: from app/services/groups/destroy_service.rb:34:in `execute'

   15: from app/models/concerns/storage/legacy_namespace.rb:109:in `rm_dir'

   14: from app/models/concerns/storage/legacy_namespace.rb:109:in `each'

   13: from app/models/concerns/storage/legacy_namespace.rb:114:in `block in rm_dir'

   12: from lib/gitlab/gitaly_client/namespace_service.rb:12:in `allow'

   11: from lib/gitlab/temporarily_allow.rb:9:in `temporarily_allow'

   10: from lib/gitlab/gitaly_client/namespace_service.rb:12:in `block in allow'

    9: from app/models/concerns/storage/legacy_namespace.rb:115:in `block (2 levels) in rm_dir'

    8: from lib/gitlab/shell.rb:170:in `mv_namespace'

    7: from lib/gitlab/gitaly_client/namespace_service.rb:40:in `rename'

    6: from lib/gitlab/gitaly_client/namespace_service.rb:46:in `gitaly_client_call'

    5: from lib/gitlab/gitaly_client.rb:167:in `call'

    4: from lib/gitlab/gitaly_client/call.rb:17:in `call'

   3: from lib/gitlab/gitaly_client/call.rb:55:in `recording_request'

    2: from lib/gitlab/gitaly_client/call.rb:18:in `block in call'

    1: from lib/gitlab/gitaly_client.rb:177:in `execute'

GRPC::Internal (13:failed proxying to secondary: rpc error: code = InvalidArgument desc = from directory gpt/many_groups_and_projects/gpt-subgroup-122 not found. debug_error_string:{"created":"@1604500203.889677008","description":"Error received from peer ipv4:10.207.120.12:3305","file":"src/core/lib/surface/call.cc","file_line":1055,"grpc_message":"failed proxying to secondary: rpc error: code = InvalidArgument desc = from directory gpt/many_groups_and_projects/gpt-subgroup-122 not found","grpc_status":13})

Results of GitLab environment info

Running Gitlab 13.5.1 with Gitaly Cluster

Possible fixes

Edited by Kevin Vogt