Better handling for merge requests with a large amount of commit
Summary
Note: This might also be better suited to the Gitaly issue tracker
One of our customers (internal) reported an issue about a high memory usage for the gitaly-git2go-v14
process:
git 24071 104 87.0 20003936 14301224 Sl 17:08:20 01:29:49 futex_wait_queue_me /opt/gitlab/embedded/bin/gitaly-git2go-v14 conflicts
Upon checking, it seems that a merge request with a large amount of commits (~2000 commits) was created for a specific project.
This was noted in the Gitaly logs:
{
"command.count": 3,
"command.cpu_time_ms": 22338032,
"command.inblock": 7446706,
"command.majflt": 187796,
"command.maxrss": 15385212,
"command.minflt": 16481399,
"command.oublock": 0,
"command.real_time_ms": 21625414,
"command.system_time_ms": 21026431,
"command.user_time_ms": 1311600,
"correlation_id": "01GJGBQKQ5EPEF0WVZ4QY3QH27",
"error": "rpc error: code = Canceled desc = conflicts: ",
"grpc.code": "Canceled",
"grpc.meta.auth_version": "v2",
"grpc.meta.client_name": "gitlab-sidekiq",
"grpc.meta.deadline_type": "unknown",
"grpc.meta.method_type": "server_stream",
"grpc.method": "ListConflictFiles",
"grpc.request.deadline": "2022-11-24T09:33:47.592",
"grpc.request.fullMethod": "/gitaly.ConflictsService/ListConflictFiles",
...
This specific method is called by MergeRequestMergeabilityCheckWorker
.
gitaly-git2go-v14
was then eventually OOM-killed by the kernel:
Wed Nov 23 05:05:17 2022] Out of memory: Kill process 5664 (gitaly) score 634 or sacrifice child
[Wed Nov 23 05:05:17 2022] Killed process 31948 (gitaly-git2go-v) total-vm:25093044kB, anon-rss:1445744kB, file-rss:0kB, shmem-rss:0kB
[Wed Nov 23 05:05:17 2022] oom_reaper: reaped process 31948 (gitaly-git2go-v), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Closing the merge reqeust via the UI is not possible as the page is hung.
The customer managed to close merge request via the rails console. However, it didn't stop MergeRequestMergeabilityCheckWorker
from triggering.
We also notice multiple instances of MergeRequestMergeabilityCheckWorker
which makes it worse as it spins-up multiple instances of gitaly-git2go-v14 conflicts
.
Steps to reproduce
The customer experienced this issue on GitLab 14.9
- Create an MR with a large amount of commit.
- Based on
git-sizer
, the project also contains large blobs of files.
- Try to load the MR
What is the current bug behavior?
gitaly-git2go-v14 conflicts
consumes a large amount of memory when trying a merge request with a large amount of commit was created.
What is the expected correct behavior?
GitLab handles the merge request with a large amount of commits properly