Projects scheduled for deletion cannot be deleted immediately — PG::QueryCanceled statement timeout cascading on deployments.project_id_convert_to_bigint

Summary

On GitLab.com, once a project has been scheduled for deletion, any subsequent attempt to delete it immediately fails. The initial scheduling succeeds (202), but every follow-up permanent-deletion request fails in the background with a Postgres statement timeout while cascading a temporary bigint-conversion foreign key on deployments. The project is returned to its pending-deletion state.

The UI reports each deletion attempt as successful, so the failure is only apparent after refreshing the page, or via the banner displayed once the project is restored. There is no self-service path to remove an affected project.

This can be reproduces with a blank project created directly in GitLab, as well as with a project left behind by a cancelled GitHub import, the latter having been the initial suspected failure route. Affected projects contain no data of any kind.

Steps to reproduce

Reliably reproducible on GitLab.com:

  1. Create a blank project in a group namespace. (A shell project left behind by a cancelled GitHub import behaves identically.)
  2. Delete the project. It is scheduled for deletion with a 30-day retention window. This step succeeds.
  3. Attempt to delete the project immediately / permanently.
  4. The UI indicates success. Refresh — the project is still present.
  5. Repeat step 3 any number of times, including from within the scheduled-for-deletion project itself. It never succeeds.
  6. Restore the project and visit it. A banner reports the underlying deletion failure.

What is the current bug behavior?

Permanent deletion silently fails every time. Banner shown on the restored project:

This project was scheduled for deletion, but failed with the following message:
PG::QueryCanceled: ERROR: canceling statement due to statement timeout
CONTEXT: SQL statement "DELETE FROM ONLY "public"."deployments"
WHERE $1 OPERATOR(pg_catalog.=) "project_id_convert_to_bigint""
The project visibility may have been made more restrictive if the parent group's
visibility changed while the deletion was scheduled.

What is the expected correct behavior?

A project scheduled for deletion can be permanently deleted. If deletion cannot complete, the UI reports the failure at the time of the request instead of indicating success.

Reproduction timeline and correlation IDs

All from a single test project on GitLab.com, for Kibana lookup:

Time (UTC) Correlation ID Action Result
2026-08-07T18:49:29.296 a2787621ccbaeabf Schedule project for deletion Success, 202 DELETE
2026-08-07T18:51:36.247 a278793e7cdeeabf Delete immediately Fails, PG::QueryCanceled (statement timeout)
2026-08-07T18:54:51.547 a2787e034eabeabf Delete immediately again Fails, project still present
2026-08-07T18:58:57.495 a27884024a4aeabf Delete from within the scheduled-for-deletion project Fails; Gitaly logs NotFound on FindDefaultBranchName

Gitaly entry from the final attempt:

FindDefaultBranchName  gitlab-web  @hashed/<hashed path>.git
gitaly-01-stor-gprd...  0.228  NotFound

This NotFound is most likely a side effect of the failed deletion rather than a cause — the repository is absent or already unlinked by the time the retry runs. Noted for completeness, since it appears consistently alongside the failures.

Analysis

The failing statement targets deployments.project_id_convert_to_bigint, the shadow column of an in-progress integer → bigint conversion, not the real project_id column. The GitLab.com schema validation report #423301 confirms a temporary cascading FK exists in production:

ExtraForeignKeys | public.fk_b9a3851b82_tmp | deployments |
  FOREIGN KEY (project_id_convert_to_bigint) REFERENCES projects(id) ON DELETE CASCADE

Every project deletion therefore cascades through two FKs on deployments. The failure occurs on the temporary one, ProjectDestroyWorker rolls back, and the project returns to pending deletion with the error persisted on the record.

Key detail for the database team: the affected projects are blank. They have zero deployments rows. A cascade delete matching zero rows should be effectively instant, so a statement timeout strongly suggests the FK check on the shadow column is not using an index and is sequentially scanning deployments in its entirety. #423301 does list bigint_idx_* indexes leading on project_id_convert_to_bigint, so the query plan needs confirming — the correlation IDs above should allow retrieval of the exact plan.

Impact

  • Any project scheduled for deletion on GitLab.com may become permanently undeletable by its owner.
  • Requires backend intervention to clear; no user-facing workaround exists.
  • The UI reporting success on each failed attempt masks the problem and invites repeated retries.
  • Affects empty projects, so it is not a data-volume problem and is likely broad in scope.

Possibly related issues

Surfaced during triage; relationship not confirmed:

  • #24262 — Some projects stuck in pending_delete (generic tracker, cause unidentified)
  • #588289 (closed) — Deletion records stuck pending when repository missing (similar class of failure, different code path)
  • #364707 — Delete imported projects after user cancelled import (explains how one of the test projects arose; not a cause)
  • #423301 — Schema validation report documenting the temporary FK

Issue description generated with AI assistance

Edited by 🤖 GitLab Bot 🤖