When someone deletes a project, it's very hard to restore it again if the deletion was accidental. The database can only be restored from a backup previously done. For the repository, it's usually removed from disk with a delay, which increases the chance of restoring it, but that is also not a guarantee.
Further details
Deleting something as big as a project should have safe-guards in place. One important one is an intermediate state where it can be moved to, in which it's not available anymore to regular users, but it can still be restored if it was removed by mistake.
This is also useful as a removal can even happen from an API request.
Proposal
Implement the concept of delayed deletion, in which projects can be "removed" but not yet deleted. So any removal operation will put the project in that place (state), and that will flag the project for removal in X days (30 7 by default, but can be configured for compliance reasons). Note: Implemented as "soft delete" under #32935 (closed)
When it is "removed" it will appear for the regular user as if it has been deleted completely. But it can still be restored by a system administrator. There should be an Admin UI screen that can list "removed" projects, when it was "removed" and when it will be deleted completely, with a button to "restore". Note: Implemented in !33502 (closed)
#344013 (closed) | A removed repository is removed from any indexing routine and is excluded from any relevant query (you shouldn't search for anything that is related to a removed project, as we do for archived ones.
#344016 (closed) | There should be the option to "delete all removed projects now" as well, which should be useful when you want to free space from disk.
#344014 (closed) | In the past this proposal was not easily possible as the repositories were named the same as their namespace/project_slug. With Hashed Storage, this is no longer an issue and we can safely delay the removal while still allowing a new project to be created with the exact same name as the previous one.
As part of overall DR strategy, Customer wants to be able to recover deleted projects. Having a time-boxed archival/recovery process would be one aspect of recovering (accidentally!) deleted projects.
This comment in the related issue suggests a similar 2-tier approach to managing the project deletion process.
@fzimmer@joshlambert: Do you see groupgeo being able to prioritize this? This possibly relates to either DR or backup-restore.
Manage implemented the first version of this in #32935 (closed) only out of customer urgency.
I think we've taken off some urgency here now that we have a first iteration out, and I think future iterations on this might be a better fit for Enablement. Let me know if you're thinking about solving this problem in a different way.
Hrm. I personally don't think this would be a good fit for groupgeo. The Geo team is focused on recoverying quickly from a failed instance.
This issue here is more about providing a self-service path to recovery in the event someone made a destructive change within the app. Making the Geo team responsible for all variations of this across the GitLab app code base would be a significant change in focus, and should probably be owned by the teams who work on this area. (They should know best how to design for these cases, etc.)
@jeremy is the problem in this instance, that no group currently owns the project/group structure? Could this be the subgroup team by chance?
Yeah, I don't wholly disagree; there's no completely natural fit at the moment.
This issue here is more about providing a self-service path to recovery in the event someone made a destructive change within the app.
I brought up Geo since it feels related to that problem space - the prevention of a undesired and destructive change. I'm not sure where we landed on project level backup-restore; I thought Geo was considering that for this particular customer, so it seemed related.
I suppose Manage:Compliance would be the only other group that could be related, but it's also not an elegant fit.
is the problem in this instance, that no group currently owns the project/group structure? Could this be the subgroup team by chance?
Subgroups is a category, but Projects isn't. I'm fine with a group owning this as a one-off, but I'm resistant to making Projects a dedicated responsibility.
The categories of Backup/Restore and Geo Replication look at a GitLab instance as a whole. It would be a change for us to think of the data at the project level. I suggest that the team responsible for creating projects should also be the team responsible for their deletion. So I am also supportive of Projects being a dedicated responsibility.
@mjang1 thanks for calling that out. The proposal doesn't mention the API (and the open MR does not affect the API) so it looks like we can focus on 1. And I'm happy to write that. But first, do you know of a good place in the documentation?
Dan Jensenmarked the checklist item When it is "trashed" it will appear for the regular user as if it has been removed completely. But it can still be restored, by a system administrator. There should be an Admin UI screen that can list "trashed" projects, when it was "trashed" and when it will be deleted completely, with a button to "restore". Note: Implemented in !33502 (closed) as completed
marked the checklist item When it is "trashed" it will appear for the regular user as if it has been removed completely. But it can still be restored, by a system administrator. There should be an Admin UI screen that can list "trashed" projects, when it was "trashed" and when it will be deleted completely, with a button to "restore". Note: Implemented in !33502 (closed) as completed
Dan Jensenchanged title from Implement a recycle-bin for deleted projects to Implement delayed deletion for projects
changed title from Implement a recycle-bin for deleted projects to Implement delayed deletion for projects
Dan Jensenchanged the descriptionCompare with previous version
@mattgonzales the spec here says that only a system administrator should be able to restore a removed project. Do you see any reason we shouldn't allow users with remove permission to also have restore permission? My thinking is that we should empower users to fix their own mistakes. (This question appeared in MR review.)
@djensen That's a great point. Yes, users should be able to restore projects as well, imo.
I think the crossed wires may be due to the "Removed projects" view and the intent behind project restoration. The project view serves to consolidate all of these removed projects in a single view for administrators (as a concept, not the role) to recover them easily; restoring a deleted project by itself should be available to anyone to "fix their own mistakes"
@mattgonzales awesome, I just created an issue for allowing non-admins to restore projects and see the "removed projects" tab of the projects dashboard: #222472 (closed)
In a previous time, it was the default that projects in our namespace were soft-deleted for one week. When this became an option in the group's UI, soft deletes were turned off by default, which I somewhat agree with since that is a precedent set by other SCMs. However, this was an unexpected change in the project delete feature. I have two proposals that I think could make this easier to manage (we're users of gitlab.com.)
Allow this setting to work for projects recursively down into subgroup trees. I'm not sure if this would only make sense in GMA's, and maybe only for the top-level group. The downside to this option is that if you're absolutely certain a project needs to be deleted immediately, you would need to disable this option for all projects under the top-level group, which could lead to an accidental permanent delete if someone else went to delete a project in the window that you disabled it for the project you're trying to permanently delete. For that reason, I'm not sure if I prefer that solution.
Keep the current behavior of the setting only working directly underneath that group, but write an API to enable/disable that option. I didn't see the addition for an API for it in this issue. The reason I think this is valuable is that users can script tooling to ensure that some desirable settings are set across their namespace. And if someone needs to immediately delete a project, they would need to explicitly disable that setting for only that group, and then perform the project delete. Much less likely to cause accidental permanent deletes.
I think option 2 makes the most sense personally. Let me know if you think this needs to be a new Issue and I can write one up.
I'm not sure we've decided how to implement immediate deletion, but we definitely want to offer something
API to enable/disable that option
Now that you've confirmed there is demand for that, here is a new issue: #238850
I think option 2 makes the most sense personally.
Yep, I agree. That's not to say we shouldn't pursue option 1 (inheritance) later, but option 2 (API) is much simpler, so we could deliver it sooner. I'll try to get that ready for development soon
I really like the 2-person approach to offering immediate deletes. If a project enters a soft-deleted state due to the delayed deletion for projects setting, I think if two people were to make a combined judgement call that a project needed to be deleted immediately then it's really on them if it was a mistake imho. Also like the Audit Events part in the issue you linked Immediate deletion by a second user#233379 (closed)
@bbutler38 excellent. I believe @mattgonzales (our Product Manager) is leaning toward the 2-person approach as well, and it makes sense to me too. I'm guessing that's what we will end up building.
Thanks again for all this feedback, can't tell you how helpful it has been.
I'm supportive of the dialogue and I don't think I have anything new or contrary to add
I am definitely leaning towards the two-person approach for deleting projects. That will likely be the second iteration following a two-person approval to override an MR (bypass compliance checks, approval rules, etc) to deploy emergency fixes.
@djensen I think you are the only active that owuld have context, but I believe we met the intent of this issue with delayed deletion for projects. I'm going to close this issue.
We can open issues for the remaining proposal items if they become evident problems to solve.
@aregnery yeah I do think we should open issues for the remaining proposal items, because some of them are important. It has actually been on my to-do list for 2 weeks to convert this to an epic and create sub-issues, I just haven't had time. Hopefully I can handle that later this week