unable to remove old project, LFS files still there

Summary

After updating my gitlab installation to 10.3.3 I tried to delete a quite large project holding a lot of lfs files which I had migrated using gitmigrate beforehand from another local gitlab repo without lfs enabled.

I'm running into ProjectDestroyWorker output of the dead job:

ActiveRecord::StatementInvalid: PG::ConnectionBad: PQsocket() can't get socket descriptor: BEGIN

Since this conversion was just a test I wanted to delete this repo and rebuild it with the most recent state of my non-lfs repo.

Steps to reproduce

Unfortunately I cannot roll back this since the old repo was in there since Gitlab 9.4.

What is the current bug behavior?

I'm unabled to delete the project. It disapears in the GUI but still seems to be in the database and even worse 16 GB of LFS data is still in there.

Actually it is one of the projects which are marked as empty during gitlab-rake gitlab:check SANITIZE=true

What is the expected correct behavior?

Project get's removed completely and also LFS data which is the most important part for me.

Results of GitLab environment info

Expand for output related to GitLab environment info 
gitlab-rake gitlab:env:info


System information
System:
Current User:	git
Using RVM:	no
Ruby Version:	2.3.5p376
Gem Version:	2.6.13
Bundler Version:1.13.7
Rake Version:	12.3.0
Redis Version:	3.2.11
Git Version:	2.14.3
Sidekiq Version:5.0.4
Go Version:	unknown


GitLab information
Version:	10.3.3
Revision:	888cf31
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	postgresql
URL:		http://git.qfs.de
HTTP Clone URL:	http://git.qfs.de/some-group/some-project.git
SSH Clone URL:	ssh://git@git.qfs.de:23/some-group/some-project.git
Using LDAP:	no
Using Omniauth:	no


GitLab Shell
Version:	5.10.2
Repository storage paths:
  • default: /var/opt/gitlab/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check 
root@eeeded838d70:/# gitlab-rake gitlab:check SANITIZE=true
Checking GitLab Shell ...

GitLab Shell version >= 5.10.2 ? ... OK (5.10.2)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:root, or git:git?
default... yes
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ...
3/11 ... ok
9/12 ... ok
3/13 ... ok
4/14 ... ok
4/15 ... ok
4/16 ... ok
9/18 ... ok
6/19 ... ok
4/22 ... ok
9/23 ... ok
9/25 ... ok
16/27 ... ok
3/28 ... ok
2/29 ... ok
4/30 ... ok
2/31 ... ok
3/35 ... ok
2/36 ... repository is empty
27/37 ... ok
3/41 ... ok
4/44 ... ok
3/46 ... ok
4/47 ... ok
27/48 ... ok
27/53 ... ok
27/54 ... ok
27/55 ... ok
27/56 ... ok
27/57 ... ok
27/58 ... ok
27/59 ... ok
27/60 ... ok
27/61 ... ok
4/62 ... ok
3/63 ... ok
2/64 ... ok
5/65 ... ok
9/66 ... ok
27/67 ... ok
9/68 ... ok
3/70 ... ok
2/72 ... repository is empty
3/73 ... ok
9/77 ... ok
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Redis available via internal API: OK


Access to /var/opt/gitlab/.ssh/authorized_keys: OK
gitlab-shell self-check successful


Checking GitLab Shell ... Finished


Checking Sidekiq ...


Running? ... yes
Number of Sidekiq processes ... 1


Checking Sidekiq ... Finished


Reply by email is disabled in config/gitlab.yml
Checking LDAP ...


LDAP is disabled in config/gitlab.yml


Checking LDAP ... Finished


Checking GitLab ...


Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
3/11 ... yes
9/12 ... yes
3/13 ... yes
4/14 ... yes
4/15 ... yes
4/16 ... yes
9/18 ... yes
6/19 ... yes
4/22 ... yes
9/23 ... yes
9/25 ... yes
16/27 ... yes
3/28 ... yes
2/29 ... yes
4/30 ... yes
2/31 ... yes
3/35 ... yes
2/36 ... yes
27/37 ... yes
3/41 ... yes
4/44 ... yes
3/46 ... yes
4/47 ... yes
27/48 ... yes
27/53 ... yes
27/54 ... yes
27/55 ... yes
27/56 ... yes
27/57 ... yes
27/58 ... yes
27/59 ... yes
27/60 ... yes
27/61 ... yes
4/62 ... yes
3/63 ... yes
2/64 ... yes
5/65 ... yes
9/66 ... yes
27/67 ... yes
9/68 ... yes
3/70 ... yes
2/72 ... yes
3/73 ... yes
9/77 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.3.5 ? ... yes (2.3.5)
Git version >= 2.7.3 ? ... yes (2.14.3)
Git user has default SSH configuration? ... yes
Active users: ... 28


Checking GitLab ... Finished

Possible fixes