Expiring branches on an empty repo causes a Gitlab::Git::Repository::NoRepository error
Sentry error: https://sentry.gitlap.com/gitlab/gitlabcom/issues/131637/
On the GPRD nodes, I am seeing a lot of these Gitaly errors when the repo is empty:
2018-03-01_00:42:29.78084 sidekiq-besteffort-01-sv-gprd sidekiq-cluster: 2018-03-01T00:42:29.780Z 9759 TID-osmke3nl0 WARN: {"context":"Job raised exception","job":{"class":"Geo::ProjectSyncWorker","args":[3036491,"2018-03-01 00:42:17 +0000"],"retry":3,"queue":"geo_project_sync","dead":false,"jid":"65dacad654e35d1056e05a44","created_at":1519864937.8940928,"enqueued_at":1519864945.4390702,"error_message":"5:GetRepoPath: not a git repository '/var/opt/gitlab/git-data-file08/repositories/redacted-namespace/redacted-project.git'","error_class":"Gitlab::Git::Repository::NoRepository","failed_at":1519864938.0025878,"retry_count":0},"jobstr":"{\"class\":\"Geo::ProjectSyncWorker\",\"args\":[3036491,\"2018-03-01 00:42:17 +0000\"],\"retry\":3,\"queue\":\"geo_project_sync\",\"dead\":false,\"jid\":\"65dacad654e35d1056e05a44\",\"created_at\":1519864937.8940928,\"enqueued_at\":1519864945.4390702,\"error_message\":\"5:GetRepoPath: not a git repository '/var/opt/gitlab/git-data-file08/repositories/namespace/project-bpm.git'\",\"error_class\":\"Gitlab::Git::Repository::NoRepository\",\"failed_at\":1519864938.0025878,\"retry_count\":0}"}
2018-03-01_00:42:29.78130 sidekiq-besteffort-01-sv-gprd sidekiq-cluster: 2018-03-01T00:42:29.781Z 9759 TID-osmke3nl0 WARN: Gitlab::Git::Repository::NoRepository: 5:GetRepoPath: not a git repository '/var/opt/gitlab/git-data-file08/repositories/namespace/project'
2018-03-01_00:42:29.78140 sidekiq-besteffort-01-sv-gprd sidekiq-cluster: 2018-03-01T00:42:29.781Z 9759 TID-osmke3nl0 WARN: /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/git/repository.rb:1340:in `rescue in gitaly_migrate'
2018-03-01_00:42:29.78146 sidekiq-besteffort-01-sv-gprd sidekiq-cluster: /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/git/repository.rb:1337:in `gitaly_migrate'
2018-03-01_00:42:29.78147 sidekiq-besteffort-01-sv-gprd sidekiq-cluster: /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/git/repository.rb:202:in `local_branches'
2018-03-01_00:42:29.78148 sidekiq-besteffort-01-sv-gprd sidekiq-cluster: /opt/gitlab/embedded/service/gitlab-rails/app/models/repository.rb:750:in `local_branches'
2018-03-01_00:42:29.78151 sidekiq-besteffort-01-sv-gprd sidekiq-cluster: /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:159:in `block in branches'
2018-03-01_00:42:29.78152 sidekiq-besteffort-01-sv-gprd sidekiq-cluster: /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/method_call.rb:36:in `measure'
2018-03-01_00:42:29.78154 sidekiq-besteffort-01-sv-gprd sidekiq-cluster: /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:159:in `branches'
2018-03-01_00:42:29.78155 sidekiq-besteffort-01-sv-gprd sidekiq-cluster: /opt/gitlab/embedded/service/gitlab-rails/app/models/repository.rb:349:in `expire_branch_cache'
2018-03-01_00:42:29.78156 sidekiq-besteffort-01-sv-gprd sidekiq-cluster: /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:159:in `block in expire_branch_cache'
2018-03-01_00:42:29.78157 sidekiq-besteffort-01-sv-gprd sidekiq-cluster: /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/method_call.rb:36:in `measure'
2018-03-01_00:42:29.78159 sidekiq-besteffort-01-sv-gprd sidekiq-cluster: /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:159:in `expire_branch_cache'
2018-03-01_00:42:29.78161 sidekiq-besteffort-01-sv-gprd sidekiq-cluster: /opt/gitlab/embedded/service/gitlab-rails/ee/app/models/ee/repository.rb:21:in `after_sync'
2018-03-01_00:42:29.78162 sidekiq-besteffort-01-sv-gprd sidekiq-cluster: /opt/gitlab/embedded/service/gitlab-rails/ee/app/services/geo/repository_sync_service.rb:46:in `expire_repository_caches'
2018-03-01_00:42:29.78163 sidekiq-besteffort-01-sv-gprd sidekiq-cluster: /opt/gitlab/embedded/service/gitlab-rails/ee/app/services/geo/repository_sync_service.rb:9:in `sync_repository'
2018-03-01_00:42:29.78164 sidekiq-besteffort-01-sv-gprd sidekiq-cluster: /opt/gitlab/embedded/service/gitlab-rails/ee/app/services/geo/base_sync_service.rb:33:in `block in execute'
2018-03-01_00:42:29.78166 sidekiq-besteffort-01-sv-gprd sidekiq-cluster: /opt/gitlab/embedded/service/gitlab-rails/ee/app/services/concerns/exclusive_lease_guard.rb:22:in `try_obtain_lease'
2018-03-01_00:42:29.78167 sidekiq-besteffort-01-sv-gprd sidekiq-cluster: /opt/gitlab/embedded/service/gitlab-rails/ee/app/services/geo/base_sync_service.rb:27:in `execute'
2018-03-01_00:42:29.78169 sidekiq-besteffort-01-sv-gprd sidekiq-cluster: /opt/gitlab/embedded/service/gitlab-rails/ee/app/workers/geo/project_sync_worker.rb:22:in `perform'
2018-03-01_00:42:29.78170 sidekiq-besteffort-01-sv-gprd sidekiq-cluster: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:188:in `execute_job'
2018-03-01_00:42:29.78171 sidekiq-besteffort-01-sv-gprd sidekiq-cluster: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:170:in `block (2 levels) in process'
2018-03-01_00:42:29.78173 sidekiq-besteffort-01-sv-gprd sidekiq-cluster: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'
2018-03-01_00:42:29.78174 sidekiq-besteffort-01-sv-gprd sidekiq-cluster: /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_status/server_middleware.rb:5:in `call'
2018-03-01_00:42:29.78175 sidekiq-besteffort-01-sv-gprd sidekiq-cluster: /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
# ls -al
total 36
drwxrwx--- 6 git git 4096 Feb 28 01:59 .
drwxrwx--- 4 git root 4096 Feb 28 01:53 ..
-rw-r--r-- 1 git git 256 Feb 28 01:59 config
-rw-r--r-- 1 git git 73 Feb 16 10:30 description
-rw-r--r-- 1 git git 0 Feb 28 01:44 FETCH_HEAD
-rw-r--r-- 1 git git 23 Feb 16 10:30 HEAD
lrwxrwxrwx 1 git git 47 Feb 16 10:30 hooks -> /opt/gitlab/embedded/service/gitlab-shell/hooks
drwxr-xr-x 2 git git 4096 Feb 16 10:30 hooks.old.1518777034
drwxr-xr-x 2 git git 4096 Feb 16 10:30 info
drwxr-xr-x 4 git git 4096 Feb 16 10:30 objects
drwxr-xr-x 4 git git 4096 Feb 16 10:30 refs
# git log
fatal: your current branch 'master' does not have any commits yet
# ls -al refs/heads
total 8
drwxr-xr-x 2 git git 4096 Feb 16 10:30 .
drwxr-xr-x 4 git git 4096 Feb 16 10:30 ..
Edited by Stan Hu