Unable to delete a project
<!---
Please read this!
Before opening a new issue, make sure to search for keywords in the issues
filtered by the "regression" or "type::bug" label:
- https://gitlab.com/gitlab-org/gitlab/issues?label_name%5B%5D=regression
- https://gitlab.com/gitlab-org/gitlab/issues?label_name%5B%5D=type::bug
and verify the issue you're about to submit isn't a duplicate.
--->
### Summary
<!-- Summarize the bug encountered concisely. -->
Some projects fail to be deleted. The deletion command is triggered with a REST API call, UI or the rails console, the outcome is the same (see error below).
If it helps at all, the projects are quite old in the current GitLab installation.
### Steps to reproduce
<!-- Describe how one can reproduce the issue - this is very important. Please use an ordered list. -->
Not sure if this is reproducible, just issue a REST API call, UI action or rails console commands:
**With curl**
- `curl -s -X DELETE -H "PRIVATE-TOKEN: $PRIVATE_TOKEN" https://git-server/api/v4/projects/ID`
**In the UI**
- Go to project page `-> Settings -> General -> Advanced -> Delete project`.
**In rails console**
```
project = Project.find_by_full_path('<project_path>')
user = User.find_by_username('<username>')
ProjectDestroyWorker.new.perform(project.id, user.id, {})
=> false
```
### What is the current *bug* behavior?
<!-- Describe what actually happens. -->
- During a cleanup, over 500 projects were deleted in the current GitLab installation.
- There are a couple of projects that cannot be deleted and it is not clear what their state is.
- The initial DELETE command happened via the REST API.
- After the HTTP call, these projects show the "No repository" message in their landing page (`namespace/project path`).
- One of them even returned a 500 error, which was "fixed" after running a `gitlab-rake cache:clear` command.
- Now any subsequent delete operation (REST API, UI, console) is failing with the error log provided below.
- Both project do not appear to show in the "pending for deletion" queue.
### What is the expected *correct* behavior?
<!-- Describe what you should see instead. -->
The projects should be deleted.
### Relevant logs and/or screenshots
<!-- Paste any relevant logs - please use code blocks (```) to format console output, logs, and code
as it's tough to read otherwise. -->
The error log is the following:
```
"message":"Deletion failed on xxx/xxx with the following message: Locking a record with unpersisted changes is not supported. Use `save` to persist the changes, or `reload` to discard them explicitly."
```
```
"exception.class":"RuntimeError","exception.message":"Locking a record with unpersisted changes is not supported. Use `save` to persist the changes, or `reload` to discard them explicitly.","exception.backtrace":["app/models/issue.rb:668:in `update_upvotes_count'","app/models/award_emoji.rb:77:in `expire_cache'","lib/gitlab/database/load_balancing/connection_proxy.rb:121:in `public_send'","lib/gitlab/database/load_balancing/connection_proxy.rb:121:in `block in write_using_load_balancer'","lib/gitlab/database/load_balancing/load_balancer.rb:127:in `block in read_write'","lib/gitlab/database/load_balancing/load_balancer.rb:198:in `retry_with_backoff'","lib/gitlab/database/load_balancing/load_balancer.rb:116:in `read_write'","lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `write_using_load_balancer'","lib/gitlab/database/load_balancing/connection_proxy.rb:72:in `transaction'","app/models/concerns/cross_database_modification.rb:92:in `block in transaction'","lib/gitlab/database/load_balancing/connection_proxy.rb:121:in `public_send'","lib/gitlab/database/load_balancing/connection_proxy.rb:121:in `block in write_using_load_balancer'","lib/gitlab/database/load_balancing/load_balancer.rb:127:in `block in read_write'","lib/gitlab/database/load_balancing/load_balancer.rb:198:in `retry_with_backoff'","lib/gitlab/database/load_balancing/load_balancer.rb:116:in `read_write'","lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `write_using_load_balancer'","lib/gitlab/database/load_balancing/connection_proxy.rb:72:in `transaction'","lib/gitlab/database.rb:369:in `block in transaction'","lib/gitlab/database.rb:368:in `transaction'","app/models/concerns/cross_database_modification.rb:83:in `transaction'","lib/gitlab/database/load_balancing/connection_proxy.rb:121:in `public_send'","lib/gitlab/database/load_balancing/connection_proxy.rb:121:in `block in write_using_load_balancer'","lib/gitlab/database/load_balancing/load_balancer.rb:127:in `block in read_write'","lib/gitlab/database/load_balancing/load_balancer.rb:198:in `retry_with_backoff'","lib/gitlab/database/load_balancing/load_balancer.rb:116:in `read_write'","lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `write_using_load_balancer'","lib/gitlab/database/load_balancing/connection_proxy.rb:72:in `transaction'","app/models/concerns/batch_destroy_dependent_associations.rb:27:in `block in destroy_dependent_associations_in_batches'","app/models/concerns/batch_destroy_dependent_associations.rb:23:in `each'","app/models/concerns/batch_destroy_dependent_associations.rb:23:in `destroy_dependent_associations_in_batches'","app/services/projects/destroy_service.rb:145:in `destroy_project_related_records'","app/services/projects/destroy_service.rb:124:in `attempt_destroy'","app/services/projects/destroy_service.rb:30:in `execute'","app/workers/project_destroy_worker.rb:20:in `perform'","(irb):7:in `\u003cmain\u003e'"],"user.username":null,"tags.program":"console","tags.locale":"en","tags.feature_category":null,"tags.correlation_id":null,"extra.project_id":55}
```
### Output of checks
#### Results of GitLab environment info
<!-- Input any relevant GitLab environment information if needed. -->
<details>
<summary>Expand for output related to GitLab environment info</summary>
<pre>
# Docker installation.
# Image: 15.11.13-ce.0
#
# gitlab-rake gitlab:env:info
```
System information
System:
Current User: git
Using RVM: no
Ruby Version: 3.0.6p216
Gem Version: 3.2.33
Bundler Version:2.3.15
Rake Version: 13.0.6
Redis Version: 6.2.11
Sidekiq Version:6.5.7
Go Version: unknown
GitLab information
Version: 15.11.13
Revision: cc3748fcd2d
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: PostgreSQL
DB Version: 13.11
URL: https://git-server
HTTP Clone URL: https://git-server/some-group/some-project.git
SSH Clone URL: git@git-server:some-group/some-project.git
Using LDAP: yes
Using Omniauth: yes
Omniauth Providers:
GitLab Shell
Version: 14.18.0
Repository storages:
- default: unix:/var/opt/gitlab/gitaly/gitaly.socket
GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell
```
</pre>
</details>
#### Results of GitLab application Check
<!-- Input any relevant GitLab application check information if needed. -->
<details>
<summary>Expand for output related to the GitLab application check</summary>
<pre>
```
# gitlab-rake gitlab:check SANITIZE=true`)
Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 14.18.0 ? ... OK (14.18.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes
Number of Sidekiq processes (cluster/worker) ... 1/1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
#
# Skipped some lines...
#
Checking LDAP ... Finished
Checking GitLab App ...
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Cable config exists? ... yes
Resque config exists? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Projects have namespace: ...
#
# All "yes", for the sake of simplicity.
#
Redis version >= 6.0.0? ... yes
Ruby version >= 2.7.2 ? ... yes (3.0.6)
Git user has default SSH configuration? ... yes
Active users: ... 200
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished
```
</pre>
</details>
### Possible fixes
<!-- If you can, link to the line of code that might be responsible for the problem. -->
issue