Make Gitaly less error prone on GDK
Like @dosuken123 reported on https://gitlab.com/gitlab-org/gitlab-ce/issues/39860#note_46274503 I started getting 503s on GDK after updating to ruby-2.3.5. I attempted to run the gitaly-ruby command separately and got the following:
alejandro@MacBook-Pro:~/gdk-ee/gitaly/ruby (master)$ bundle exec bin/gitaly-ruby 40615 /var/folders/2k/8lwrq3g52wb09g7scny656nr0000gn/T/gitaly-ruby139628073/socket
bundler: failed to load command: bin/gitaly-ruby (bin/gitaly-ruby)
LoadError: incompatible library version - /Users/alejandro/gdk-ee/gitaly/assembly/ruby/vendor/bundle/ruby/2.3.0/gems/rugged-0.26.0/lib/rugged/rugged.bundle
/Users/alejandro/gdk-ee/gitaly/assembly/ruby/vendor/bundle/ruby/2.3.0/gems/rugged-0.26.0/lib/rugged.rb:10:in `require'
/Users/alejandro/gdk-ee/gitaly/assembly/ruby/vendor/bundle/ruby/2.3.0/gems/rugged-0.26.0/lib/rugged.rb:10:in `rescue in <top (required)>'
/Users/alejandro/gdk-ee/gitaly/assembly/ruby/vendor/bundle/ruby/2.3.0/gems/rugged-0.26.0/lib/rugged.rb:6:in `<top (required)>'
/Users/alejandro/gdk-ee/gitaly/assembly/ruby/lib/gitlab/git.rb:2:in `require'
/Users/alejandro/gdk-ee/gitaly/assembly/ruby/lib/gitlab/git.rb:2:in `<top (required)>'
/Users/alejandro/gdk-ee/gitaly/assembly/ruby/lib/gitaly_server.rb:3:in `require_relative'
/Users/alejandro/gdk-ee/gitaly/assembly/ruby/lib/gitaly_server.rb:3:in `<top (required)>'
bin/gitaly-ruby:7:in `require_relative'
bin/gitaly-ruby:7:in `<top (required)>'
So it seems the problem is likely caused by library version changes between the time the gem was built and now. Now, even though I'm not entirely sure why this error happened I did solve it without nuking everything (which is what people have been doing) and I identified some things we could improve in our setup:
- Notice that we use a vendored gem path, unlike what we do with the rest of the GDK components (gitlab-rails and gitlab-shell) where we use the global gems (unless the user specifies a BUNDLER_PATH). We hardcoded
BUNDLER_PATH=vendor/bundle
because of problems with source installations (see !264 (merged)). However, there's no reason why we should vendor on GDK. SOLUTION: havemake install
use the much nicer (since it doesn't pollute the bundler settings)--deployment
flag for source installations, which is what we do on gitlab-rails (see https://docs.gitlab.com/ce/install/installation.html#install-gems). Then, change gitlab-rails to usemake install
instead ofmake
. - Unlike all other components, we don't always
bundle install
ongdk update
: in ourMakefile
, that task depends onruby/Gemfile
andruby/Gemfile.lock
so if they're not updated,bundle install
won't run. This is nice for our development and testing cycle but it may leave us open to errors like the one above. SOLUTION: do what the rest of the components do and alwaysbundle install
ongdk update
.
/cc @gl-gitaly
Edited by Alejandro Rodríguez