Repository never correctly removed

Summary

Repository check doesn't remove a failed removed repository.

We have one project with this symptoms :

  • wget https://mygitlab/my-repo return 404
  • git clone https://mygitlab/my-repo.git work fine
  • git clone git@mygitlab:my-repo.gitwork fine
  • in cloned repository, git push and git pull work fine
  • git push to git@mygitlab:copy-of-my-repo.git work fine
  • in gitlab IHM, rename copy-of-my-repo to my-repo return this error: There is already a repository with that name on disk and The project is still being deleted. Please try again later.
  • on https://mygitlab/admin/projects/my-repo I can trigger a new repository check, but that return no error and my-repo is never clean.

How to realy remove this repository ?

Steps to reproduce

I doesn't reproduce it, but I suppose this scenario could work :

  • create a new repository with files
  • request to remove it with IHM
  • break sidkiq job at good time

Example Project

I haven't right on gitlab.com to kill sidekiq process :)

What is the current bug behavior?

  • https://mygitlab/my-repo return a 404, it's impossible to manage this repository
  • repository check doesn't report error
  • repository is never removed.

What is the expected correct behavior?

IMHO the repository check must launch a new sidkiq job to really clean this repository, or remove repository job must be in one big transaction.

Relevant logs and/or screenshots

  • Screenshot_2019-05-02_A0599_ce_dev___Projects___Admin_Area

logs when I GET on https://mygitlab/my-repo :

Started GET "/A0599/ce_dev/edit" for 127.0.0.1 at 2019-05-02 10:19:49 +0200
Processing by ProjectsController#edit as HTML
  Parameters: {"namespace_id"=>"A0599", "id"=>"ce_dev"}
Filter chain halted as :project rendered or redirected
Completed 404 Not Found in 110ms (Views: 91.9ms | ActiveRecord: 2.5ms | Elasticsearch: 0.0ms)

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info 
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:env:info)


System information
System:
Proxy:          no_proxy: 127.0.0.1,localhost,ice,ice-dev,ldc1i781.server.lan
http_proxy: http://srvproxy.appli:8080
https_proxy: http://srvproxy.appli:8080
Current User:   git
Using RVM:      no
Ruby Version:   2.5.3p105
Gem Version:    2.7.6
Bundler Version:1.16.6
Rake Version:   12.3.2
Redis Version:  3.2.12
Git Version:    2.18.1
Sidekiq Version:5.2.5
Go Version:     unknown


GitLab information
Version:        11.9.7-ee
Revision:       782806f
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     postgresql
DB Version:     9.6.11
URL:            https://git-prd.server.lan
HTTP Clone URL: https://git-prd.server.lan/some-group/some-project.git
SSH Clone URL:  git@git-prd.server.lan:some-group/some-project.git
Elasticsearch:  no
Geo:            no
Using LDAP:     yes
Using Omniauth: yes
Omniauth Providers:


GitLab Shell
Version:        8.7.1
Repository storage paths:
  • default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check 
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true)


gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes
Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... Server: ldapmain
LDAP authentication... Success
LDAP users with access to your GitLab server (only showing the first 100 results)


[...]


Checking LDAP ... Finished


Checking GitLab App ...


Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... no
Try fixing it:
sudo chown -R git /var/opt/gitlab/gitlab-rails/uploads
sudo find /var/opt/gitlab/gitlab-rails/uploads -type f -exec chmod 0644 {} ;
sudo find /var/opt/gitlab/gitlab-rails/uploads -type d -not -path /var/opt/gitlab/gitlab-rails/uploads -exec chmod 0700 {} ;
For more information see:
doc/install/installation.md in section "GitLab"
Please fix the error above and rerun the checks.
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
3/1 ... yes
[...]
1343/3123 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.3.5 ? ... yes (2.5.3)
Git version >= 2.18.0 ? ... yes (2.18.1)
Git user has default SSH configuration? ... no
Try fixing it:
mkdir ~/gitlab-check-backup-1556784528
sudo mv /var/opt/gitlab/.ssh/id_rsa.pub ~/gitlab-check-backup-1556784528
sudo mv /var/opt/gitlab/.ssh/id_rsa ~/gitlab-check-backup-1556784528
For more information see:
doc/ssh/README.md in section "SSH on the GitLab server"
Please fix the error above and rerun the checks.
Active users: ... 596
Elasticsearch version 5.6 - 6.x? ... skipped (elasticsearch is disabled)


Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished
Edited by 🤖 GitLab Bot 🤖