Resources on staging are not being permanently deleted
Summary
Ref: gitlab-com/gl-infra/production-engineering#25178 (comment 1809992319)
Test resource subgroups on staging that have been marked for deletion and permanently deleted are still there and remain marked for deletion.
Looking at one of the groups that were scheduled to be destroyed 2024-03-07: https://staging.gitlab.com/gitlab-qa-sandbox-group-2/qa-test-2024-03-04-16-38-21-00f32f4d25bff041
Tracing the deletion of this group, correlation ID: d3efa85853921096099d23fcf14c8d97
.
We can see the failure from Sidekiq:
Error as seen in logs:
PG::NotNullViolation: ERROR: null value in column "namespace_id" of relation "work_item_dates_sources" violates not-null constraint
DETAIL: Failing row contains (2024-03-04 16:44:40.402793+00, 2024-03-04 16:44:40.402793+00, 19569541, null, t, t, null, null, null, null, null, null, null, null).
CONTEXT: SQL statement "UPDATE ONLY "public"."work_item_dates_sources" SET "namespace_id" = NULL WHERE $1 OPERATOR(pg_catalog.=) "namespace_id""
Searching for "work_item_dates_sources" turns up this epic #440415 (closed) which is possibly related?