Redis broke down, no pipelines starting, MR not updating on push ...
Summary
On our omnibus Gitlab EE installation, Redis has stopped working. As a result, everything besides pushing does not seem to work. Merge requests are not update, pipelines are not started ... It started this night according to the logs I can find. This is urgent, as none of the gitlab functionality is currently working.
Restart did not help, on the admin page it showed an update was available so I tried this update to 9.1.4. The upgrade process broke down halfway, so there is also that.
Steps to reproduce
Stopped working without any user action (in the middle of the night)
What is the current bug behavior?
Everything that should happen after push actions does not.
What is the expected correct behavior?
Normal operation
Relevant logs and/or screenshots
Upon doing the restart command:
ok: run: gitaly: (pid 20599) 1s
ok: run: gitlab-monitor: (pid 20611) 0s
ok: run: gitlab-workhorse: (pid 20615) 0s
ok: run: logrotate: (pid 20626) 1s
ok: run: nginx: (pid 20633) 0s
ok: run: node-exporter: (pid 20640) 1s
ok: run: postgres-exporter: (pid 20645) 0s
ok: run: postgresql: (pid 20653) 1s
ok: run: prometheus: (pid 20662) 0s
ok: run: redis: (pid 20673) 0s
ok: run: redis-exporter: (pid 20679) 1s
timeout: run: sidekiq: (pid 25694) 116311s, got TERM
ok: run: unicorn: (pid 20747) 1s
Sidekiq log showing issues:
Upgrade log failure (perhaps somewhat unrelated, as the problem happened apart from it, but the error here seems to be sidekiq/Redis related, so possibly useful...):
Output of checks
Results of GitLab environment info
System information System: Debian 8.8 Proxy: no Current User: git Using RVM: no Ruby Version: 2.3.3p222 Gem Version: 2.6.6 Bundler Version:1.13.7 Rake Version: 10.5.0 Redis Version: 3.2.5 Git Version: 2.11.1 Sidekiq Version:4.2.7
GitLab information Version: 9.1.4-ee Revision: cd361c2 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: postgresql DB Version: 9.6.1 URL: ****** HTTP Clone URL: /some-group/some-project.git SSH Clone URL: git@:some-group/some-project.git Elasticsearch: yes Geo: no Using LDAP: no Using Omniauth: yes Omniauth Providers: google_oauth2, bitbucket
GitLab Shell Version: 5.0.2 Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Checking GitLab Shell ...
GitLab Shell version >= 5.0.2 ? ... OK (5.0.2) Repo base directory exists? default... yes Repo storage directories are symlinks? default... no Repo paths owned by git:git? default... yes Repo paths access is drwxrws---? default... yes hooks directories in repos are links: ... 4/2 ... ok 4/3 ... ok 4/5 ... ok 4/7 ... ok 4/8 ... ok 4/9 ... ok 4/10 ... ok 4/11 ... ok 4/12 ... ok 4/13 ... ok 4/14 ... ok 4/15 ... ok 4/16 ... ok 4/17 ... ok 4/18 ... ok 4/19 ... ok 4/20 ... ok 4/21 ... ok 4/22 ... ok 4/23 ... ok 4/24 ... ok 4/25 ... ok 4/26 ... ok 4/28 ... ok 4/29 ... ok 4/30 ... ok 4/32 ... ok 4/33 ... repository is empty 4/35 ... ok 4/36 ... ok 4/38 ... ok 16/39 ... ok 4/40 ... ok 17/41 ... ok Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Check GitLab API access: OK Access to /var/opt/gitlab/.ssh/authorized_keys: OK Send ping to redis server: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Sidekiq ...
Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Reply by email ...
Reply by email is disabled in config/gitlab.yml
Checking Reply by email ... Finished
Checking LDAP ...
LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab ...
Git configured with autocrlf=input? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config outdated? ... no Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory setup correctly? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) projects have namespace: ... 4/2 ... yes 4/3 ... yes 4/5 ... yes 4/7 ... yes 4/8 ... yes 4/9 ... yes 4/10 ... yes 4/11 ... yes 4/12 ... yes 4/13 ... yes 4/14 ... yes 4/15 ... yes 4/16 ... yes 4/17 ... yes 4/18 ... yes 4/19 ... yes 4/20 ... yes 4/21 ... yes 4/22 ... yes 4/23 ... yes 4/24 ... yes 4/25 ... yes 4/26 ... yes 4/28 ... yes 4/29 ... yes 4/30 ... yes 4/32 ... yes 4/33 ... yes 4/35 ... yes 4/36 ... yes 4/38 ... yes 16/39 ... yes 4/40 ... yes 17/41 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.1.0 ? ... yes (2.3.3) Your git bin path is "/opt/gitlab/embedded/bin/git" Git version >= 2.7.3 ? ... yes (2.11.1) Active users: 13 Elasticsearch version 5.1.x? ... no, you have 5.2.2
Checking GitLab ... Finished