Cannot stat Git binary

Summary

After upgrading from GitLab 15.8.5 to 15.11.11

  1. 'Cannot stat Git binary' error seen in the webservice logs.
  2. /var/opt/gitlab/gitaly/run/gitaly-32303/git-exec-3534582711.d/git points to a version that doesn't exist (v2.38) obraz obraz.

Steps to reproduce

  1. Deploy Gitlab 15.8.5 using GET (Gitlab Environment Toolkit)
  2. Upgrade to GitLab 15.11.11 using GET

What is the current bug behavior?

Error is seen in the webservice logs and 500 error is seen in the browser when browsing repositories.

What is the expected correct behavior?

No errors are seen.

Relevant logs

Gitlab::Git::CommandError (13:getting Git version: cannot stat Git binary: stat /var/opt/gitlab/gitaly/run/gitaly-32253/git-exec-2331936717.d/git: no such file or directory.):
  lib/gitlab/git/wraps_gitaly_errors.rb:15:in `rescue in wrapped_gitaly_errors'
  lib/gitlab/git/wraps_gitaly_errors.rb:6:in `wrapped_gitaly_errors'
  lib/gitlab/git/repository.rb:1162:in `uncached_has_local_branches?'
  lib/gitlab/git/repository.rb:194:in `block in has_local_branches?'
  lib/gitlab/utils/strong_memoize.rb:34:in `strong_memoize'
  lib/gitlab/git/repository.rb:193:in `has_local_branches?'
  app/models/repository.rb:600:in `has_visible_content?'
  lib/gitlab/repository_cache_adapter.rb:95:in `block (2 levels) in cache_method_asymmetrically'
  lib/gitlab/repository_cache.rb:44:in `fetch_without_caching_false'
  lib/gitlab/repository_cache_adapter.rb:190:in `block (2 levels) in cache_method_output_asymmetrically'
  lib/gitlab/safe_request_store.rb:12:in `fetch'
  lib/gitlab/repository_cache.rb:25:in `fetch'
  lib/gitlab/repository_cache_adapter.rb:189:in `block in cache_method_output_asymmetrically'
  lib/gitlab/utils/strong_memoize.rb:34:in `strong_memoize'
  lib/gitlab/repository_cache_adapter.rb:203:in `block in memoize_method_output'
  lib/gitlab/repository_cache_adapter.rb:212:in `no_repository_fallback'
  lib/gitlab/repository_cache_adapter.rb:202:in `memoize_method_output'
  lib/gitlab/repository_cache_adapter.rb:188:in `cache_method_output_asymmetrically'
  lib/gitlab/repository_cache_adapter.rb:94:in `block in cache_method_asymmetrically'
  app/models/repository.rb:574:in `empty?'
  app/graphql/resolvers/blobs_resolver.rb:30:in `resolve'
  lib/gitlab/graphql/present/field_extension.rb:18:in `resolve'
  lib/gitlab/graphql/tracers/timer_tracer.rb:20:in `trace'
  lib/gitlab/graphql/generic_tracing.rb:48:in `with_labkit_tracing'
  lib/gitlab/graphql/generic_tracing.rb:38:in `platform_trace'
  lib/gitlab/graphql/tracers/logger_tracer.rb:14:in `trace'
  lib/gitlab/graphql/tracers/metrics_tracer.rb:13:in `trace'
  lib/gitlab/graphql/tracers/application_context_tracer.rb:23:in `trace'
  lib/gitlab/graphql/tracers/timer_tracer.rb:20:in `trace'
  lib/gitlab/graphql/generic_tracing.rb:48:in `with_labkit_tracing'
  lib/gitlab/graphql/generic_tracing.rb:38:in `platform_trace'
  lib/gitlab/graphql/tracers/logger_tracer.rb:14:in `trace'
  lib/gitlab/graphql/tracers/metrics_tracer.rb:13:in `trace'
  lib/gitlab/graphql/tracers/application_context_tracer.rb:23:in `trace'
  lib/gi… [message truncated due to size]

Details of package version

root@gl-gitaly-1:~# dpkg-query -l "gitlab-*"
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version       Architecture Description
+++-==============-=============-============-============================================================
un  gitlab-ce      <none>        <none>       (no description available)
hi  gitlab-ee      15.11.11-ee.0 amd64        GitLab Enterprise Edition (including NGINX, Postgres, Redis)
un  gitlab-fips    <none>        <none>       (no description available)

Environment details

  • Operating System: Ubuntu
  • Installation Target, remove incorrect values:
    • VM: GCP and K8S (cloud native hybrid)
  • Installation Type, remove incorrect values:
    • Upgrade from version 15.8.5
  • Is this a single or multiple node installation? Single
  • Resources
    • CPU: REPLACE-WITH-DETAILS
    • Memory total: REPLACE-WITH-DETAILS

Configuration details

Provide the relevant sections of `/etc/gitlab/gitlab.rb` 
root@gl-gitaly-1:~# grep -v -e '^#' -e '^$' /etc/gitlab/gitlab.rb
postgresql['enable'] = false
redis['enable'] = false
puma['enable'] = false
sidekiq['enable'] = false
gitlab_workhorse['enable'] = false
prometheus['enable'] = false
alertmanager['enable'] = false
grafana['enable'] = false
gitlab_exporter['enable'] = false
nginx['enable'] = false
gitlab_kas['enable'] = false
gitlab_rails['rake_cache_clear'] = false
gitlab_rails['auto_migrate'] = false
gitlab_rails['internal_api_url'] = 'https://XXXXXXXXXXXXXXXXXX'
node_exporter['listen_address'] = '0.0.0.0:9100'
gitaly['configuration'] = {
  listen_addr: '0.0.0.0:8075',
  pack_objects_cache: {
    enabled: true,
  },
  gitaly_ruby: {
    num_workers: 2
  },
  prometheus_listen_addr: '0.0.0.0:9236',
  auth: {
    token: 'XXXXXXXXXXXXXXXXXX',
  },
  storage: [
    {
      name: 'default',
      path: '/var/opt/gitlab/git-data/repositories'
    },
    {
      name: 'storage1',
      path: '/var/opt/gitlab/git-data/storage1/repositories'
    }
  ],
}
custom_confs = Dir.glob(File.join("/etc/gitlab/", "gitlab.gitaly.*.rb"))
custom_confs.each { |conf|
  from_file conf
}
Edited by Tomasz Brozek