Projects unable to be deleted due to statement timeouts
We've received a number of reports that projects get into a state where their repository has been deleted on disk, but the database record remains: https://gitlab.com/gitlab-com/support-forum/issues/3127.
The best clue so far comes from messages such as:
This project was scheduled for deletion, but failed with the following message: PG::QueryCanceled: ERROR: canceling statement due to statement timeout CONTEXT: SQL statement "DELETE FROM ONLY "public"."ci_build_trace_sections" WHERE $1 OPERATOR(pg_catalog.=) "section_name_id"" : DELETE FROM "projects" WHERE "projects"."id" = 5562300Sentry error: https://sentry.gitlap.com/gitlab/gitlabcom/issues/141346/
In looking at our database, we can have 767 failures, 360 of which are due to that statement timeout above:
irb(main):034:0> unique = Project.where.not(delete_error: nil).count
=> 760
irb(main):035:0> unique = Project.where.not(delete_error: nil).where("delete_error LIKE '%DELETE FROM%'").count
=> 360The others are attributed to other error messages:
| Count | Error |
|---|---|
| 360 | "PG::QueryCanceled: ERROR: canceling statement due to statement timeout\nCONTEXT: SQL statement "DELETE FROM ONLY "public"."ci_build_trace_sections" WHERE $1 OPERATOR(pg_catalog.=) "section_name_id""\n: DELETE FROM "projects" WHERE "projects"."id" = x" |
| 126 | Failed to open TCP connection to registry.gitlab.com:443 (getaddrinfo: Name or service not known) |
| 111 | 2:Call dropped by load balancing policy |
| 36 | 14:Endpoint read failed |
| 30 | 4:Deadline Exceeded |
| Unknown | 2:GitCommand: start [/opt/gitlab/embedded/bin/git --git-dir /var/opt/gitlab/git-data/repositories/namespace/project.git for-each-ref --format=%(refname)%00%(objectname)%00%(contents:subject)%00%(authorname)%00%(authoremail)%00%(authordate:iso-strict)%00%(committername)%00%(committeremail)%00%(committerdate:iso-strict) --sort=refname refs/heads]: fork/exec /opt/gitlab/embedded/bin/git: resource temporarily unavailable |
| 18 | Failed to remove project repository. Please try again or contact administrator. |
| 16 | Failed to remove some tags in project container registry. Please try again or contact administrator. |
| 9 | undefined method `destroy' for nil:NilClass |
| 9 | 7:permission denied |
| 8 | 2:process spawn timed out after 10s |
| 6 | undefined method `merge_requests' for nil:NilClass |
| 6 | Validation failed: Lfs object already exists in project |
| 5 | undefined method `lfs_objects' for nil:NilClass |
| 5 | Failed to open TCP connection to registry.gitlab.com:443 (getaddrinfo: No address associated with hostname) |
| 5 | 12:unknown service gitaly.RepositoryService |
| 2 | Failed to destroy the record |
| Unknown | Input/output error @ rb_file_s_rename - (/var/opt/gitlab/git-data-file11/repositories/namespace/project.git, /var/opt/gitlab/git-data-file11/repositories/namespace/project+41546612+deleted.git) |
| 1 | 4:rpc error: code = DeadlineExceeded desc = context deadline exceeded |
| 1 | 14:Connect Failed |
Edited by Stan Hu