Unable to sync GitLab CE repository into temporary directory due to lock file
I was looking for a repository to test the snapshot feature in #338 (closed), but I happened to spot the CE repository claiming to be out of sync when, in fact, it did had the latest commit on disk (b8a848304edc50ec4d4dfbb895dc3c16896c8e10):
irb(main):011:0> failed = Geo::ProjectRegistryFinder.new.find_failed_project_registries
irb(main):012:0> failed.last.project.repository.path
=> "/var/opt/gitlab/git-data-file08/repositories/gitlab-org/gitlab-ce.git"
irb(main):013:0> failed.last
=> #<Geo::ProjectRegistry id: 7655925, project_id: 13083, last_repository_synced_at: "2018-05-02 04:32:23", last_repository_successful_sync_at: nil, created_at: "2018-05-02 03:06:33", resync_repository: true, esync_wiki: false, last_wiki_synced_at: "2018-05-02 03:06:33", last_wiki_successful_sync_at: "2018-05-02 03:06:33", repository_retry_count: 10, repository_retry_at: "2018-05-02 06:25:29", force_to_redownload__epository: nil, wiki_retry_count: nil, wiki_retry_at: nil, force_to_redownload_wiki: false, last_repository_sync_failure: "Error syncing repository: 2:Fetching remote geo fa...", last_wiki_sync_failure: nil, ast_repository_verification_failure: nil, last_wiki_verification_failure: nil, repository_verification_checksum_sha: nil, wiki_verification_checksum_sha: nil>
irb(main):014:0> puts failed.last.last_repository_sync_failure
Error syncing repository: 2:Fetching remote geo failed: error: cannot lock ref 'refs/keep-around/93e8cbd19082c61de1c319e3ec9f92bd4c31835e': Unable to create '/var/opt/gitlab/git-data/repositories/@geo-temporar/gitlab-org/gitlab-ce.git/./refs/keep-around/93e8cbd19082c61de1c319e3ec9f92bd4c31835e.lock': File exists.
Another git process seems to be running in this repository, e.g.
an editor opened by 'git commit'. Please make sure all processes
are terminated then try again. If it still fails, a git process
may have crashed in this repository earlier:
remove the file manually to continue.
error: cannot lock ref 'refs/keep-around/d46c7e33983650c5d6079bc5dcacee87ee2a5917': Unable to create '/var/opt/gitlab/git-data/repositories/@geo-temporary/gitlab-org/gitlab-ce.git/./refs/keep-around/d46c7e3398650c5d6079bc5dcacee87ee2a5917.lock': File exists.
Another git process seems to be running in this repository, e.g.
an editor opened by 'git commit'. Please make sure all processes
are terminated then try again. If it still fails, a git process
may have crashed in this repository earlier:
remove the file manually to continue.
It's almost as though two processes are trying to redownload this repository at the same time in the same place.
/cc: @vsizov