Deleted Projects not being properly deleted
Summary
Deleting a project causes the project not to be entirely deleted.
Steps to reproduce
- Create project
- Set for adjourned deletion
- Wait for adjourned deletion to occur
- Attempt to load project - it will load
Example Project
(If possible, please create an example project here on GitLab.com that exhibits the problematic behavior, and link to it here in the bug report)
(If you are using an older version of GitLab, this will also determine whether the bug is fixed in a more recent version)
What is the current bug behavior?
Project still loads after apparent deletion but with no data:
What is the expected correct behavior?
Project does not load after deletion.
Relevant logs and/or screenshots
Output of checks
(If you are reporting a bug on GitLab.com, write: This bug happens on GitLab.com)
Possible fixes
When deletion is attempted, SQL errors are raised that prevent the full deletion sequence. Example:
[ gprd ] production> p.delete_error
=> "PG::QueryCanceled: ERROR: canceling statement due to statement timeout\nCONTEXT: SQL statement \"DELETE FROM ONLY \"public\".\"ci_pipelines_config\" WHERE $1 OPERATOR(pg_catalog.=) \"pipeline_id\"\"\n"
Designs
- Show closed items
Relates to
- gitlab-org/foundations/general-discussion #17283
Activity
-
Newest first Oldest first
-
Show all activity Show comments only Show history only
- Max Woolf added devopsmanage groupcompliance typebug labels
added devopsmanage groupcompliance typebug labels
- Max Woolf closed
closed
- Developer
from slack thread, couldn't find the relevant sentry entry but similar looking ones:
- Cleveland Bledsoe Jr reopened
reopened
- Developer
We're experiencing this with another customer project, so I think it is worth re-opening this. Details are in https://gitlab.com/gitlab-com/support/internal-requests/-/issues/2745
DeleteStoredFilesWorker
is failing due to an object not found. - 🤖 GitLab Bot 🤖 mentioned in issue #232358 (closed)
mentioned in issue #232358 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue #233314 (closed)
mentioned in issue #233314 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue #235069 (closed)
mentioned in issue #235069 (closed)
- Dan Jensen changed the description
Compare with previous version changed the description
- Contributor
Collapse replies
- Contributor
Note: Technically this probably belongs to groupproject management, but groupcompliance has been responsible for a series of project deletion enhancements recently (related to adjourned/soft deletion). I think it makes sense for groupcompliance to be responsible for this issue even though it appears in the context of ordinary project deletion as well.
Edited by Dan Jensen - Dan Jensen added backend workflowplanning breakdown labels
added backend workflowplanning breakdown labels
- Dan Jensen added Next Up label
added Next Up label
- Max Woolf set weight to 3
set weight to 3
- Max Woolf added workflowscheduling label and removed workflowplanning breakdown label
added workflowscheduling label and removed workflowplanning breakdown label
- 🤖 GitLab Bot 🤖 added sectiondev label
added sectiondev label
- 🤖 GitLab Bot 🤖 mentioned in issue #237807 (closed)
mentioned in issue #237807 (closed)
- Maintainer
Some of the work I'm doing on this issue: https://gitlab.com/gitlab-org/gitlab/-/issues/24644 appears to be related.
Collapse replies - Contributor
@allison.browne nice, thanks for calling that out. Is that one a result of query timeouts? In other words, do you think these 2 issues are duplicative? (If so, I'd want to close this one because yours is much further along.)
- Contributor
+Like. +Subscribe.
I'm following this thread since this is in workflowscheduling for groupcompliance and I'd love to know if we need to plan for this in %13.5
- Contributor
@mattgonzales oops! Glad you found this on your own. Yeah, should almost certainly be in the %13.5 plan
- Contributor
@djensen It is done!
- Maintainer
I'm not sure my MR's will solve all problems with this since I'm focused on speeding up the ci parts and making sure they eventually delete via retries, but I am hiding projects where destruction is started from the projects index.
- Contributor
@allison.browne cool, we'll keep this issue open then
By the way, your MR to hide projects represents a step forward on the checklist in #24866 (closed), so I just noted it there
- 🤖 GitLab Bot 🤖 mentioned in issue #240838 (closed)
mentioned in issue #240838 (closed)
- Matt Gonzales (ex-GitLab) changed milestone to %13.5
changed milestone to %13.5
- Matt Gonzales (ex-GitLab) added workflowready for development label and removed Next Up workflowscheduling labels
added workflowready for development label and removed Next Up workflowscheduling labels
- Dan Jensen mentioned in issue #210001 (closed)
mentioned in issue #210001 (closed)
- 🤖 GitLab Bot 🤖 added [deprecated] Accepting merge requests label
added [deprecated] Accepting merge requests label
- Matt Gonzales (ex-GitLab) added priority2 label
added priority2 label
- Matt Gonzales (ex-GitLab) mentioned in issue gitlab-org/manage/general-discussion#17272
mentioned in issue gitlab-org/manage/general-discussion#17272
- Matt Gonzales (ex-GitLab) added Deliverable label
added Deliverable label
- Contributor
Moving this to %13.6 as a stretch goal. Please see the groupcompliance planning issue.
- Matt Gonzales (ex-GitLab) changed milestone to %13.6
changed milestone to %13.6
- 🤖 GitLab Bot 🤖 added SLOMissed label
added SLOMissed label
- Matt Gonzales (ex-GitLab) mentioned in issue gitlab-org/manage/general-discussion#17283
mentioned in issue gitlab-org/manage/general-discussion#17283
- Matt Gonzales (ex-GitLab) marked this issue as related to gitlab-org/manage/general-discussion#17283
marked this issue as related to gitlab-org/manage/general-discussion#17283
- Matt Gonzales (ex-GitLab) mentioned in issue #270200 (closed)
mentioned in issue #270200 (closed)
- Matt Gonzales (ex-GitLab) added priority3 label and removed priority2 label
- Dan Jensen mentioned in issue #19856 (closed)
mentioned in issue #19856 (closed)
- Max Woolf added workflowin dev label and removed workflowready for development label
added workflowin dev label and removed workflowready for development label
- 🤖 GitLab Bot 🤖 removed [deprecated] Accepting merge requests label
removed [deprecated] Accepting merge requests label
Collapse replies Yeah, so currently when the
Projects::DestroyService
runs, it destroys eachCi::Build
in turn becauseproject has_many :ci_builds, dependent: :destroy
This causes the following SQL query to run on project destruction:
DELETE FROM "ci_builds" WHERE "ci_builds"."id" = 1 DELETE FROM "ci_builds" WHERE "ci_builds"."id" = 2 DELETE FROM "ci_builds" WHERE "ci_builds"."id" = 3 DELETE FROM "ci_builds" WHERE "ci_builds"."id" = N
Ideally, we should just delete all of the ci builds attached to the project by using
dependent: :delete_all
. Since we're trying to delete data, there's no compelling reason I can see to run validations on the data.DELETE FROM "ci_builds" WHERE "ci_builds"."type" = $1 AND "ci_builds"."project_id"
Would be executed instead if we changed the
depdendent:
method.I've just spotted the comment in
Project
though which may well throw this idea out the window...# Ci::Build objects store data on the file system such as artifact files and # build traces. Currently there's no efficient way of removing this data in # bulk that doesn't involve loading the rows into memory. As a result we're # still using `dependent: :destroy` here.
- Maintainer
@mwoolf, This issue covers that: https://gitlab.com/gitlab-org/gitlab/-/issues/24644
However full completion of the issue was blocked on another issue, until a few days ago.
Great, thanks @allison.browne.
I think we'll leave this issue alone for now then until #24644 is complete to see if there are further issues that need addressing./cc @djensen
- Contributor
- Contributor
@mattgonzales sounds good, done
- Contributor
@djensen Looks like this is still blocked due to https://gitlab.com/gitlab-org/gitlab/-/issues/24644 which is currently workflowin dev and has recent activity/
1
- Max Woolf mentioned in merge request !47503 (closed)
mentioned in merge request !47503 (closed)
- Dan Jensen changed milestone to %13.8
changed milestone to %13.8
- Max Woolf added workflowready for development label and removed workflowin dev label
added workflowready for development label and removed workflowin dev label
- 🤖 GitLab Bot 🤖 added [deprecated] Accepting merge requests label
added [deprecated] Accepting merge requests label
- Matt Gonzales (ex-GitLab) mentioned in issue gitlab-org/manage/general-discussion#17294
mentioned in issue gitlab-org/manage/general-discussion#17294
- Dan Jensen mentioned in issue gitlab-org/quality/triage-reports#1018 (closed)
mentioned in issue gitlab-org/quality/triage-reports#1018 (closed)