Deprecate ListDirectories RPC

This RPC is only used in cleanup rake tasks, as far as I can tell. E.g. https://gitlab.com/gitlab-org/gitlab-ce/blob/master/lib/tasks/gitlab/cleanup.rake#L7

This is problematic for Gitaly HA because we there will be directories on multiple different Gitaly servers and we don't want to directly expose those Gitaly servers.

We should take a look at the things that use ListDirectories, and reimplement whatever it is they do inside Gitaly. In some cases it might be automated cleanup; in other cases we might need a new RPC. In the latter case that RPC should express what we want to happen, and leave the actual work to Gitaly.

mentioned in issue #1805 (closed)

Hi @jacobvosmaer-gitlab,

Please add labels to your issue, this aids categorization and locating issues in the future.

Thanks for your help!

---

You are welcome to help improve this comment.

added auto updated label

added devopscreate groupgitaly labels

added [deprecated] Accepting merge requests label

removed [deprecated] Accepting merge requests label

We need to address this before we can ship Praefect to GitLab.com

changed milestone to %12.4

added Deliverable backstage [DEPRECATED] + 1 deleted label

added to epic &842 (closed)

added [deprecated] Accepting merge requests label

changed epic to &1877 (closed)

The crux of the issue is that the cleanup RPC removes non hashed storage left over namespaces or repositories. In other words, if I've removed a repository form GitLab called zj-gitlab/foo, than if we find zj-gitlab/foo.git on disk, we removed it with this rake task. I think it would suffice to remove this rake task.

The other rake task that uses this, would be to remove namespace left overs on disk, same story. Would never scale, but was ported anyway.

Concrete proposal:

• Remove the gitlab:cleanup:dirs rake task and documentation
• Remove the gitlab:cleanup:repos rake task and documentation
• Remove the StorageService::ListDirectories RPC from Rails
• Remove the StorageService.ListDirectories() from Gitaly+Proto

@jramsay I think it's OK to remove the rake tasks, even before 13.0, what is your opinion?

@zj-gitlab should we do gitlab#24633 (closed) in 12.4?

Sure, though we should have a way to clean a Gitaly storage post HashedStorage migration. Would you agree? In general, if there's a +moved, +orphaned, or a +deleted directory somewhere and it's stale for 30 days, should it be removed? Although I'm not sure about touching data like this, I'd rather ask a admin to clean it manually, but not sure what your opinion is about that @jramsay.

mentioned in merge request gitlab-foss!31934 (closed)

@zj-gitlab that seems likely, although we should probably give some warning and check with Support and the SREs to make sure they are being used.

Asked in the internal slack:

1. https://gitlab.slack.com/archives/C4Y5DRKLK/p1568969056092700
2. https://gitlab.slack.com/archives/CB3LSMEJV/p1568968316424800

mentioned in issue gitlab#24633 (closed)

added workflowready for development label

marked this issue as related to gitlab#24633 (closed)

assigned to @johncai

removed [deprecated] Accepting merge requests label

mentioned in merge request !1533 (closed)

mentioned in merge request gitlab!18224 (merged)

mentioned in merge request !1544 (merged)

Rescheduling to %12.5, as we can't close this issue earlier if we'd like to maintain forward compat

changed milestone to %12.5

changed weight to 1

added workflowin review label and removed workflowready for development label

Closing as !1544 (merged) got merged a while ago, cc @johncai.

closed

mentioned in issue gitlab#36091 (closed)