sidekiq crash
I use gitlab from the version ~6.0 installed from source. When new version released I manually upgrade the gitlab. But when I upgraded to ~ 8.4 (do not remember exact version) the activity stop update. I tried to wait on new release (because it was not a big issue for me) and hope that it will be fixed with new version. But it does not happened.
I investigate that problem with sidekiq.
Here is the logs:
2016-03-27T20:32:23.601Z 16938 TID-osz8v3vzk INFO: Booting Sidekiq 4.0.1 with redis options {:url=>"redis://localhost:6379", :namespace=>"resque:gitlab"}
2016-03-27T20:32:23.609Z 16938 TID-osz8v3vzk INFO: Cron Jobs - add job with name: stuck_ci_builds_worker
2016-03-27T20:32:29.164Z 16938 TID-osz8v3vzk INFO: Running in ruby 2.1.2p95 (2014-05-08 revision 45877) [x86_64-linux]
2016-03-27T20:32:29.164Z 16938 TID-osz8v3vzk INFO: See LICENSE and the LGPL-3.0 for licensing details.
2016-03-27T20:32:29.164Z 16938 TID-osz8v3vzk INFO: Upgrade to Sidekiq Pro for more features and support: http://sidekiq.org
2016-03-27T20:34:38.061Z 16938 TID-osz8v3vzk INFO: Shutting down
2016-03-27T20:34:38.088Z 16938 TID-osz9kszyw INFO: Scheduler exiting...
2016-03-27T20:34:38.099Z 16938 TID-osz8v3vzk INFO: Terminating quiet workers
2016-03-27T20:34:38.120Z 16938 TID-osz9kt1jk INFO: Scheduler exiting...
Sidekiq shut down gracefully.
2016-03-27T20:35:08.873Z 17308 TID-orp8qjd8k INFO: Booting Sidekiq 4.0.1 with redis options {:url=>"redis://localhost:6379", :namespace=>"resque:gitlab"}
2016-03-27T20:35:08.882Z 17308 TID-orp8qjd8k INFO: Cron Jobs - add job with name: stuck_ci_builds_worker
2016-03-27T20:35:15.372Z 17308 TID-orp8qjd8k INFO: Running in ruby 2.1.2p95 (2014-05-08 revision 45877) [x86_64-linux]
2016-03-27T20:35:15.372Z 17308 TID-orp8qjd8k INFO: See LICENSE and the LGPL-3.0 for licensing details.
2016-03-27T20:35:15.372Z 17308 TID-orp8qjd8k INFO: Upgrade to Sidekiq Pro for more features and support: http://sidekiq.org
2016-03-28T00:00:21.733Z 17308 TID-orp9g99h4 StuckCiBuildsWorker JID-b1261bc6cc8fa09211f8f629 INFO: start
2016-03-28T00:00:21.745Z 17308 TID-orp9g96vs INFO: Cron Jobs - add job with name: stuck_ci_builds_worker
2016-03-28T00:00:21.802Z 17308 TID-orp9g99h4 StuckCiBuildsWorker JID-b1261bc6cc8fa09211f8f629 INFO: done: 0.069 sec
*** Error in `sidekiq 4.0.1 gitlab [0 of 5 busy]': munmap_chunk(): invalid pointer: 0x00007f1b184e5490 ***- General information about gitlab
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true
Checking GitLab Shell ...
GitLab Shell version >= 2.6.11 ? ... OK (2.6.11)
Repo base directory exists? ... yes
Repo base directory is a symlink? ... no
Repo base owned by git:git? ... yes
Repo base access is drwxrws---? ... yes
hooks directories in repos are links: ...
5/1 ... ok
7/3 ... ok
8/5 ... ok
5/8 ... ok
8/9 ... ok
5/10 ... repository is empty
8/12 ... ok
5/14 ... ok
9/15 ... ok
5/16 ... ok
5/17 ... ok
8/19 ... repository is empty
7/20 ... ok
3/21 ... ok
7/22 ... repository is empty
3/23 ... ok
3/24 ... ok
3/25 ... ok
7/26 ... ok
5/27 ... ok
5/28 ... ok
5/29 ... ok
5/30 ... ok
5/31 ... ok
9/32 ... ok
3/33 ... ok
5/34 ... ok
10/35 ... ok
6/36 ... ok
3/38 ... ok
9/39 ... ok
5/42 ... ok
10/43 ... ok
5/44 ... ok
11/45 ... ok
9/47 ... ok
4/48 ... ok
5/49 ... ok
5/50 ... repository is empty
10/53 ... ok
9/54 ... ok
9/55 ... ok
9/56 ... ok
9/57 ... ok
9/58 ... ok
10/59 ... ok
8/61 ... ok
3/63 ... ok
12/64 ... ok
12/65 ... ok
Running /home/git/gitlab-shell/bin/check
Check GitLab API access: OK
Check directories and files:
/home/git/repositories: OK
/home/git/.ssh/authorized_keys: OK
Test redis-cli executable: redis-cli 2.8.17
Send ping to redis server: PONG
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Sidekiq ...
Running? ... yes
Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Reply by email ...
Reply by email is disabled in config/gitlab.yml
Checking Reply by email ... Finished
Checking LDAP ...
LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab ...
Git configured with autocrlf=input? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config outdated? ... no
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory setup correctly? ... yes
Init script exists? ... yes
Init script up-to-date? ... yes
projects have namespace: ...
5/1 ... yes
7/3 ... yes
8/5 ... yes
5/8 ... yes
8/9 ... yes
5/10 ... yes
8/12 ... yes
5/14 ... yes
9/15 ... yes
5/16 ... yes
5/17 ... yes
8/19 ... yes
7/20 ... yes
3/21 ... yes
7/22 ... yes
3/23 ... yes
3/24 ... yes
3/25 ... yes
7/26 ... yes
5/27 ... yes
5/28 ... yes
5/29 ... yes
5/30 ... yes
5/31 ... yes
9/32 ... yes
3/33 ... yes
5/34 ... yes
10/35 ... yes
6/36 ... yes
3/38 ... yes
9/39 ... yes
5/42 ... yes
10/43 ... yes
5/44 ... yes
11/45 ... yes
9/47 ... yes
4/48 ... yes
5/49 ... yes
5/50 ... yes
10/53 ... yes
9/54 ... yes
9/55 ... yes
9/56 ... yes
9/57 ... yes
9/58 ... yes
10/59 ... yes
8/61 ... yes
3/63 ... yes
12/64 ... yes
12/65 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.1.0 ? ... yes (2.1.2)
Your git bin path is "/usr/bin/git"
Git version >= 2.7.3 ? ... no
Try fixing it:
Update your git to a version >= 2.7.3 from 2.1.4
Please fix the error above and rerun the checks.
Active users: 3
Checking GitLab ... Finished!!! NOTICE
I was manually restarted sidekiq - and that why it show success check. But when sidekiq down it give me the following lines during checking:
Checking Sidekiq ...
Running? ... no
Try fixing it:
sudo -u git -H RAILS_ENV=production bin/background_jobs start
For more information see:
doc/install/installation.md in section "Install Init Script"
see log/sidekiq.log for possible errors
Please fix the error above and rerun the checks.
- Information about system
# sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production
System information
System: Debian 8.3
Current User: git
Using RVM: no
Ruby Version: 2.1.2p95
Gem Version: 2.2.2
Bundler Version:1.7.0
Rake Version: 10.5.0
Sidekiq Version:4.0.1
GitLab information
Version: 8.6.1
Revision: a96d737
Directory: /home/git/gitlab
DB Adapter: mysql2
URL: http://gitlab.home
HTTP Clone URL: http://gitlab.home/some-group/some-project.git
SSH Clone URL: git@gitlab.home:some-group/some-project.git
Using LDAP: no
Using Omniauth: no
GitLab Shell
Version: 2.6.11
Repositories: /home/git/repositories/
Hooks: /home/git/gitlab-shell/hooks/
Git: /usr/bin/git