Bundler not found on upgrade
ZD: https://gitlab.zendesk.com/agent/tickets/16011
Summary
On upgrading to GitLab CE form 8.4.3 to 8.5 the upgrade fails when backup is being created with "bundler not found".
Steps to reproduce
Considering latest release is 8.5 the upgrade took place by running apt-get install gitlab-ce.
Expected behavior
A successfully completed upgrade output.
Relevant logs and/or screenshots
Console output https://gist.github.com/cheald/6002dbe275a379a5f85a
and
root@gitlab:~# rvm use 2.3.0@no-bundler --create
ruby-2.3.0 - #gemset created /usr/local/rvm/gems/ruby-2.3.0@no-bundler
ruby-2.3.0 - #generating no-bundler wrappers..........
Using /usr/local/rvm/gems/ruby-2.3.0 with gemset no-bundler
root@gitlab:~# gitlab-rake gitlab:backup:create
/opt/gitlab/embedded/lib/ruby/site_ruby/2.1.0/rubygems/dependency.rb:318:in `to_specs': Could not find 'bundler' (>= 0.a) among 19 total gem(s) (Gem::LoadError)
Checked in 'GEM_PATH=/usr/local/rvm/gems/ruby-2.3.0@no-bundler:/usr/local/rvm/gems/ruby-2.3.0@global', execute `gem env` for more information
from /opt/gitlab/embedded/lib/ruby/site_ruby/2.1.0/rubygems/dependency.rb:327:in `to_spec'
from /opt/gitlab/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_gem.rb:64:in `gem'
from /opt/gitlab/embedded/bin/bundle:22:in `<main>'Output of checks
Results of GitLab Application Check
# gitlab-rake gitlab:check SANITIZE=true
Checking GitLab Shell ...
GitLab Shell version >= 2.6.10 ? ... OK (2.6.10)
Repo base directory exists? ... yes
Repo base directory is a symlink? ... no
Repo base owned by git:git? ... yes
Repo base access is drwxrws---? ... yes
hooks directories in repos are links: ...
7/1 ... ok
7/2 ... ok
7/3 ... ok
7/4 ... ok
7/5 ... ok
7/6 ... ok
3/7 ... ok
3/8 ... ok
7/9 ... ok
4/10 ... ok
4/11 ... ok
4/12 ... ok
4/13 ... ok
4/14 ... ok
4/15 ... ok
8/16 ... ok
7/17 ... ok
12/18 ... ok
13/19 ... ok
9/20 ... ok
9/21 ... ok
9/22 ... ok
9/23 ... ok
9/24 ... ok
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Check directories and files:
/mnt/storage/git-data/repositories: OK
/var/opt/gitlab/.ssh/authorized_keys: OK
Test redis-cli executable: redis-cli 2.8.21
Send ping to redis server: PONG
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Sidekiq ...
Running? ... yes
Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Reply by email ...
Reply by email is disabled in config/gitlab.yml
Checking Reply by email ... Finished
Checking LDAP ...
LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab ...
Git configured with autocrlf=input? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config outdated? ... no
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory setup correctly? ... no
Try fixing it:
sudo chmod 0750 /var/opt/gitlab/gitlab-rails/uploads
For more information see:
doc/install/installation.md in section "GitLab"
Please fix the error above and rerun the checks.
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
projects have namespace: ...
7/1 ... yes
7/2 ... yes
7/3 ... yes
7/4 ... yes
7/5 ... yes
7/6 ... yes
3/7 ... yes
3/8 ... yes
7/9 ... yes
4/10 ... yes
4/11 ... yes
4/12 ... yes
4/13 ... yes
4/14 ... yes
4/15 ... yes
8/16 ... yes
7/17 ... yes
12/18 ... yes
13/19 ... yes
9/20 ... yes
9/21 ... yes
9/22 ... yes
9/23 ... yes
9/24 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.1.0 ? ... yes (2.1.8)
Your git bin path is "/opt/gitlab/embedded/bin/git"
Git version >= 1.7.10 ? ... yes (2.6.2)
Active users: 7
Checking GitLab ... FinishedResults of GitLab Environment Info
# gitlab-rake gitlab:env:info
System information
System: Ubuntu 14.04
Current User: git
Using RVM: no
Ruby Version: 2.1.8p440
Gem Version: 2.5.1
Bundler Version:1.10.6
Rake Version: 10.5.0
Sidekiq Version:4.0.1
GitLab information
Version: 8.5.0
Revision: a513e09
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: postgresql
URL: https://xxxx.domain.com
HTTP Clone URL: https://xxxx.domain.com/some-group/some-project.git
SSH Clone URL: git@xxxx.domain.com:some-group/some-project.git
Using LDAP: no
Using Omniauth: yes
Omniauth Providers: github
GitLab Shell
Version: 2.6.10
Repositories: /mnt/storage/git-data/repositories
Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks/
Git: /opt/gitlab/embedded/bin/gitPossible fixes
It looks like executing apt-get install gitlab-ce will use the current RVM's gemset when running the upgrade scripts.
When running gitlab-rakein an RVM env without bundler it fails unless we're using:
`GEM_PATH=/opt/gitlab/embedded/lib/ruby/gems/2.1.0/ gitlab-rake gitlab:check SANITIZE=true``
Something like GEM_PATH=/opt/gitlab/ruby/gems gitlab_rake ... will probably address the issue.
@dblessing this is the issue @sytses mention through support. /cc @rspeicher