GitLab threw errors during 'git push' during Gitaly Upgrades
During the gitaly stage of the deploy 12.1.201907070821-7d840e4bbeb.65977070264: https://ops.gitlab.net/gitlab-com/gl-infra/deployer/pipelines/64026 Some users reported issue performing a git push operation.
% git push
Counting objects: 3, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 973 bytes | 973.00 KiB/s, done.
Total 3 (delta 2), reused 0 (delta 0)
remote: /opt/gitlab/embedded/lib/ruby/2.6.0/psych.rb:577:in `initialize': No such file or directory @ rb_sysopen - /opt/gitlab/embedded/service/gitaly-ruby/gitlab-shell/config.yml (Errno::ENOENT)
remote: from /opt/gitlab/embedded/lib/ruby/2.6.0/psych.rb:577:in `open'
remote: from /opt/gitlab/embedded/lib/ruby/2.6.0/psych.rb:577:in `load_file'
remote: from /opt/gitlab/embedded/service/gitaly-ruby/gitlab-shell/lib/gitlab_config.rb:66:in `legacy_config'
remote: from /opt/gitlab/embedded/service/gitaly-ruby/gitlab-shell/lib/gitlab_config.rb:59:in `fetch_from_legacy_config'
remote: from /opt/gitlab/embedded/service/gitaly-ruby/gitlab-shell/lib/gitlab_config.rb:32:in `log_level'
remote: from /opt/gitlab/embedded/service/gitaly-ruby/gitlab-shell/lib/gitlab_logger.rb:132:in `<top (required)>'
remote: from /opt/gitlab/embedded/service/gitaly-ruby/gitlab-shell/lib/gitlab_metrics.rb:2:in `require_relative'
remote: from /opt/gitlab/embedded/service/gitaly-ruby/gitlab-shell/lib/gitlab_metrics.rb:2:in `<top (required)>'
remote: from /opt/gitlab/embedded/service/gitaly-ruby/gitlab-shell/lib/gitlab_custom_hook.rb:3:in `require_relative'
remote: from /opt/gitlab/embedded/service/gitaly-ruby/gitlab-shell/lib/gitlab_custom_hook.rb:3:in `<top (required)>'
remote: from /opt/gitlab/embedded/service/gitaly-ruby/git-hooks/../gitlab-shell/hooks/pre-receive:18:in `require_relative'
remote: from /opt/gitlab/embedded/service/gitaly-ruby/git-hooks/../gitlab-shell/hooks/pre-receive:18:in `<main>'
To gitlab.com:gitlab-com/runbooks.git
! [remote rejected] jts/put-rules-in-gke -> jts/put-rules-in-gke (pre-receive hook declined)
error: failed to push some refs to 'git@gitlab.com:gitlab-com/runbooks.git'Similar problems also happened during the merge of an MR: https://gitlab.com/gitlab-com/dev-resources/merge_requests/1467
This deploy included an upgrade of Gitaly, we went FROM version 1.49, TO 1.51: v1.49.0...v1.51.0
It was noted by @johncai that this may be related to: !1293 (merged)
Utilize this issue to see if we can track down the root cause of the above failure and create the necessary issues to deal with this problem to prevent clientside errors in the future.