Unable to create './refs/remotes/upstream/localprep.lock': File exists.

Summary

Noticed via email stating that mygitrepo mirroring is failing into GitLab.

It looks like mirroring isn't working in mygitrepo (and hasn't since yesterday). I just got an email saying that mirroring was turned off because it failed too many times.

Repository mirroring on mygroup/mygitrepo has been paused due to too many failures. The last failure was:

2:Fetching remote upstream failed: error: cannot lock ref 'refs/remotes/upstream/localprep': Unable to create '/var/opt/gitlab/git-data/repositories/@hashed/27/47/2700112233445566778899aabbccddeeff496b06851333d2d59ab6d863225848.git/./refs/remotes/upstream/localprep.lock': File exists.

Steps to reproduce

Setup repository mirroring on an active-active gitlab-ee cluster with an nfs shared backend, (specifically AWS EFS, I know this is discouraged, we're waiting for active-active gitlab to move off this solution to comply with high availability policy), Mirror Repository

Example Project

issue with self-hosted gitlab

What is the current bug behavior?

"refs/remotes/upstream/localprep.lock" in a mirrored repository can exist when the job tries to make this file. (I couldn't find a background job correlated with this event, I am happy to delve into relevant logs if the specific details to find can be asked for, I don't expect there is much more I can provide of value to this specific error)

What is the expected correct behavior?

I expect gitlab to handle this error case with some recovery option that can cleanly deal with the conflict (if I should manually delete this file and retry let me know, but if all that needs to happen is the file be deleted, I would ask for gitlab to just do that automatically when this failure happens)

Relevant logs and/or screenshots

(from an email notification)

Repository mirroring on mygroup/mygitrepo has been paused due to too many failures. The last failure was:

2:Fetching remote upstream failed: error: cannot lock ref 'refs/remotes/upstream/localprep': Unable to create '/var/opt/gitlab/git-data/repositories/@hashed/27/47/2700112233445566778899aabbccddeeff496b06851333d2d59ab6d863225848.git/./refs/remotes/upstream/localprep.lock': File exists.

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info 
# gitlab-rake gitlab:env:info

System information
System:		Ubuntu 18.04
Proxy:		no
Current User:	git
Using RVM:	no
Ruby Version:	2.5.3p105
Gem Version:	2.7.9
Bundler Version:1.17.3
Rake Version:	12.3.2
Redis Version:	3.2.12
Git Version:	2.21.0
Sidekiq Version:5.2.7
Go Version:	unknown


GitLab information
Version:	11.11.0-ee
Revision:	deb6f84e91f
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	PostgreSQL
DB Version:	9.6.9
URL:		https://gitlab.mydomain.example.com
HTTP Clone URL:	https://gitlab.mydomain.example.com/some-group/some-project.git
SSH Clone URL:	git@gitlab.mydomain.example.com:some-group/some-project.git
Elasticsearch:	yes
Geo:		no
Using LDAP:	no
Using Omniauth:	yes
Omniauth Providers: saml


GitLab Shell
Version:	9.1.0
Repository storage paths:
  • default: /var/opt/gitlab/git-data/repositories (EFS)
  • storage1: /srv/gitlab/storage1/repositories/repositories (Gitaly SPOF, only used for repositories too big for EFS) GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check 

gitlab-rake gitlab:check


Checking GitLab subtasks ...


Checking GitLab Shell ...


GitLab Shell: ... GitLab Shell version >= 9.1.0 ? ... OK (9.1.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Redis available via internal API: OK


Access to /var/opt/gitlab/.ssh/authorized_keys: OK
gitlab-shell self-check successful


Checking GitLab Shell ... Finished


Checking Gitaly ...


Gitaly: ... default ... OK
storage1 ... OK


Checking Gitaly ... Finished


Checking Sidekiq ...


Sidekiq: ... Running? ... yes
Number of Sidekiq processes ... 1


Checking Sidekiq ... Finished


Checking Incoming Email ...


Incoming Email: ... Checking Reply by email ...


IMAP server credentials are correct? ... yes
Init.d configured correctly? ... skipped
MailRoom running? ... skipped


Checking Reply by email ... Finished


Checking Incoming Email ... Finished


Checking LDAP ...


LDAP: ... LDAP is disabled in config/gitlab.yml


Checking LDAP ... Finished


Checking GitLab App ...


Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
(all "yes")
Redis version >= 2.8.0? ... yes
Ruby version >= 2.5.3 ? ... yes (2.5.3)
Git version >= 2.21.0 ? ... yes (2.21.0)
Git user has default SSH configuration? ... yes
Active users: ... 59
Elasticsearch version 5.6 - 6.x? ... yes (6.3.1)


Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished

Possible fixes

such code is likely to touch localprep.lock although this may be used via a git interface