Gitlab CI / Forks and so on ... not working since no disk space

Summary

Hi there,

I'm actually running a community edition of gitlab on a dedicated server. We installed it using LVM for gitlab, and here was the problem :

/dev/mapper/vg-gitlabservices 276G 254G 21G 93% /var/opt/gitlab

We got it full 2 days ago, i expended it then due to many gitlab errors, but since this we're still having many problems appearing (CI not working --> Not launching since the full disk, Forks taking about 5 or 6 hours .... and so on, we're discovering more problems each hours).

Steps to reproduce

Install gitlab and mount a Logical Volume on /var/opt/gitlab, use gitlab to make it full. I restarted Gitlab, but nothing has changed.

What is the current bug behavior?

Ci are not launching anymore, forks takes a lot of time, and there is each hours more process waiting in Sidekiq. I see 25 running since 2 days, but they are not moving.

What is the expected correct behavior?

Gitlab should be working as usually after a restart.

Relevant logs and/or screenshots

Capture_d_écran_2019-09-12_à_10.14.21

On fork :

2019-09-12_08:14:14.38686 time="2019-09-12T10:14:14+02:00" level=error msg=error correlation_id=aaaTnGclXe1 error="badgateway: failed after 0s: context canceled" method=GET uri=/admin/sidekiq/stats
2019-09-12_08:15:20.37990 time="2019-09-12T08:15:20Z" level=info msg="finished unary call with code NotFound" correlation_id=UCFcELzYtK6 error="rpc error: code = NotFound desc = GetRepoPath: not a git repository '/var/opt/gitlab/git-data/repositories/eve-blabla-it-ch/deb-repository.git'" grpc.code=NotFound grpc.meta.auth_version=v2 grpc.meta.client_name=gitlab-web grpc.method=FindDefaultBranchName grpc.request.deadline="2019-09-12T08:15:30Z" grpc.request.fullMethod=/gitaly.RefService/FindDefaultBranchName grpc.request.glProjectPath=eve-blabla-it-ch/deb-repository grpc.request.glRepository=project-542 grpc.request.repoPath=eve-blabla-it-ch/deb-repository.git grpc.request.repoStorage=default grpc.request.topLevelGroup=eve-blabla-it-ch grpc.service=gitaly.RefService grpc.start_time="2019-09-12T08:15:20Z" grpc.time_ms=0.122 peer.address=@ span.kind=server system=grpc

Results of GitLab environment info

System information
System:         Debian 9.9
Current User:   git
Using RVM:      no
Ruby Version:   2.5.3p105
Gem Version:    2.7.9
Bundler Version:1.17.3
Rake Version:   12.3.2
Redis Version:  3.2.12
Git Version:    2.21.0
Sidekiq Version:5.2.7
Go Version:     unknown

GitLab information
Version:        11.11.0
Revision:       3e8ca2fb781
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     PostgreSQL
DB Version:     9.6.11
URL:            https://xxxxxxxxxxx
HTTP Clone URL: https://xxxxxxxxxxx/some-group/some-project.git
SSH Clone URL:  git@xxxxxxxxxxx:some-group/some-project.git
Using LDAP:     no
Using Omniauth: yes
Omniauth Providers: google_oauth2

GitLab Shell
Version:        9.1.0
Repository storage paths:
- default:      /var/opt/gitlab/git-data/repositories
GitLab Shell path:              /opt/gitlab/embedded/service/gitlab-shell
Git:            /opt/gitlab/embedded/bin/git

Thanks if you can help me out !

Edited by Vincent COFFIN