Gitlab CI / Forks and so on ... not working since no disk space
Summary
Hi there,
I'm actually running a community edition of gitlab on a dedicated server. We installed it using LVM for gitlab, and here was the problem :
/dev/mapper/vg-gitlabservices 276G 254G 21G 93% /var/opt/gitlab
We got it full 2 days ago, i expended it then due to many gitlab errors, but since this we're still having many problems appearing (CI not working --> Not launching since the full disk, Forks taking about 5 or 6 hours .... and so on, we're discovering more problems each hours).
Steps to reproduce
Install gitlab and mount a Logical Volume on /var/opt/gitlab, use gitlab to make it full. I restarted Gitlab, but nothing has changed.
What is the current bug behavior?
Ci are not launching anymore, forks takes a lot of time, and there is each hours more process waiting in Sidekiq. I see 25 running since 2 days, but they are not moving.
What is the expected correct behavior?
Gitlab should be working as usually after a restart.
Relevant logs and/or screenshots
On fork :
2019-09-12_08:14:14.38686 time="2019-09-12T10:14:14+02:00" level=error msg=error correlation_id=aaaTnGclXe1 error="badgateway: failed after 0s: context canceled" method=GET uri=/admin/sidekiq/stats
2019-09-12_08:15:20.37990 time="2019-09-12T08:15:20Z" level=info msg="finished unary call with code NotFound" correlation_id=UCFcELzYtK6 error="rpc error: code = NotFound desc = GetRepoPath: not a git repository '/var/opt/gitlab/git-data/repositories/eve-blabla-it-ch/deb-repository.git'" grpc.code=NotFound grpc.meta.auth_version=v2 grpc.meta.client_name=gitlab-web grpc.method=FindDefaultBranchName grpc.request.deadline="2019-09-12T08:15:30Z" grpc.request.fullMethod=/gitaly.RefService/FindDefaultBranchName grpc.request.glProjectPath=eve-blabla-it-ch/deb-repository grpc.request.glRepository=project-542 grpc.request.repoPath=eve-blabla-it-ch/deb-repository.git grpc.request.repoStorage=default grpc.request.topLevelGroup=eve-blabla-it-ch grpc.service=gitaly.RefService grpc.start_time="2019-09-12T08:15:20Z" grpc.time_ms=0.122 peer.address=@ span.kind=server system=grpc
Results of GitLab environment info
System information
System: Debian 9.9
Current User: git
Using RVM: no
Ruby Version: 2.5.3p105
Gem Version: 2.7.9
Bundler Version:1.17.3
Rake Version: 12.3.2
Redis Version: 3.2.12
Git Version: 2.21.0
Sidekiq Version:5.2.7
Go Version: unknown
GitLab information
Version: 11.11.0
Revision: 3e8ca2fb781
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: PostgreSQL
DB Version: 9.6.11
URL: https://xxxxxxxxxxx
HTTP Clone URL: https://xxxxxxxxxxx/some-group/some-project.git
SSH Clone URL: git@xxxxxxxxxxx:some-group/some-project.git
Using LDAP: no
Using Omniauth: yes
Omniauth Providers: google_oauth2
GitLab Shell
Version: 9.1.0
Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories
GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell
Git: /opt/gitlab/embedded/bin/git
Thanks if you can help me out !