Resources on staging remain after being permanently deleted
Summary
When trying to permanently delete groups and projects on staging (passing permanently_remove and full_path), the API returns a 202 accepted but the resource is not deleted. This has affected our E2E resource cleanup scripts and at least one E2E spec.
Steps to reproduce
Try to permanently delete this subgroup: https://staging.gitlab.com/gitlab-e2e-sandbox-group-6/e2e-test-2025-04-18-05-32-46-4b64a1d82c1c0db6 with
DELETE https://staging.gitlab.com/api/v4/groups/22989516?permanently_remove=true&full_path=gitlab-e2e-sandbox-group-6%2Fe2e-test-2025-04-18-05-32-46-4b64a1d82c1c0db6
It returns a 202 but the resource remains.
Example Project
What is the current bug behavior?
The resource is not deleted.
What is the expected correct behavior?
The resource is deleted.
Relevant logs and/or screenshots
By looking for the correlation id of the call in Sentry I've found:
PG::ForeignKeyViolation: ERROR: update or delete on table "projects" violates foreign key constraint "fk_rails_1fb7e812da" on table "ai_troubleshoot_job_events" (ActiveRecord::InvalidForeignKey)
DETAIL: Key (id)=(16206962) is still referenced from table "ai_troubleshoot_job_events".Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
(For installations with omnibus-gitlab package run and paste the output of: `sudo gitlab-rake gitlab:env:info`) (For installations from source run and paste the output of: `sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)
Results of GitLab application Check
Expand for output related to the GitLab application check
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true)(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true)(we will only investigate if the tests are passing)
Possible fixes
Based on the error found, these two MRs may be related: