"find_commit" Gitaly N+1 when fetching commit information on MR
We're currently having N+1 calls for Gitaly find_commit
endpoint when presenting discussions:
Steps to reproduce
- Create a MR
- Comment on a diff file
- Outdate this diff file (change it and push)
- You'll be able to see a new
find_commit
call for each new outdated file
Seems like we could be batching these requests using Gitlab::Git::Commit.batch_by_oid
with BatchLoader
, similarly as we lazy load Blobs.
Designs
- Show closed items
Relates to
Activity
-
Newest first Oldest first
-
Show all activity Show comments only Show history only
- Author Contributor
cc @smcgivern
- Andrew Newdigate mentioned in issue gitaly#1129 (closed)
mentioned in issue gitaly#1129 (closed)
- Contributor
Thanks @oswaldo, I'm glad the new performance bar was helpful! Shall we put this on 11.0 for now?
- Sean McGivern added backend label
added backend label
- Sean McGivern changed milestone to %11.0
changed milestone to %11.0
- Author Contributor
I'm glad the new performance bar was helpful!
Indeed it is! :)
Shall we put this on 11.0 for now?
I see you already did it, thanks.
- Andrew Newdigate added maintenancerefactor label
added maintenancerefactor label
- Developer
FYI @zj worked on other performance issues on this code in the past:
- https://gitlab.com/gitlab-org/gitlab-ce/issues/37432
- https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/16967
This would be an easy win too, to reduce this problem further: https://gitlab.com/gitlab-org/gitlab-ce/issues/43916
- Contributor
@oswaldo - @andrewn pointed to this MR elsewhere: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/5554
If you look at that, we appear to be doing thousands of Gitaly requests for the serialiser, too. Is that still to do with the discussions? Should we have two issues for those, or just use this one to tackle both?
- Andrew Newdigate mentioned in issue gitaly#1055 (closed)
mentioned in issue gitaly#1055 (closed)
- Andrew Newdigate marked this issue as related to gitaly#1055 (closed)
marked this issue as related to gitaly#1055 (closed)
- Author Contributor
@smcgivern Wow, I can see a few
find_commit
calls for a local MR on.json?serializer=widget
endpoint, but not thousands. Do you mind sharing details on this MR you're looking at?This seems pretty much related though, I can handle it in the same issue if that's fine.
Edited by Oswaldo Ferreira - Author Contributor
I see it's from https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/5554, but I can't see the profiler bar, is there something I can do to enable it?
- Contributor
@oswaldo do you see it on GitLab.com normally? p + b should work.
- Author Contributor
It worked, thanks.
This seems to have the same root cause. I think we can tackle it here as well.Edited by Oswaldo Ferreira - Author Contributor
I was partially wrong on https://gitlab.com/gitlab-org/gitlab-ce/issues/45190#note_68864387. I'm not being able to reproduce the same N+1 of the Discussion tab (this issue) on the
.json
endpoint. The cause of https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/5554 massivefind_commit
calls onjson
endpoint still needs further investigation. - Oswaldo Ferreira changed the description
changed the description
- Contributor
I've got the strong suspicion that this happens on the notes. The way I've tested this, is just requesting endpoints in a loop and meanwhile scraping with Prometheus. I'll try and find more data today and update this issue to why this happens
- Contributor
@oswaldo is it possible it's to do with the setting we have enabled on this project, where all discussions must be resolved before merge?
- Author Contributor
I've got the strong suspicion that this happens on the notes. The way I've tested this, is just requesting endpoints in a loop and meanwhile scraping with Prometheus
is it possible it's to do with the setting we have enabled on this project, where all discussions must be resolved before merge?
Tried it as well, doesn't seem to be the cause either :/
Would it be related to a legacy diff note behavior? This seem to be an old MR
- Contributor
@oswaldo you could probably hack in some Gitaly call tracing to the
GitalyClient
and ask someone to run through the MR profiler on production or staging to see what happens. (Just like this: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/lib/gitlab/profiler.rb#L101) - Contributor
@oswaldo and @smcgivern, doing puts debugging with
caller[0..30]
leads me to believe https://gitlab.com/gitlab-org/gitlab-ce/issues/43894 is a big issue. My understanding is that all banzai stuff is cached somewhere, but none the less, the initial call will do N + 1 calls and with LRU enabled the 'initial' call might be done more than once. 2 - Oswaldo Ferreira assigned to @oswaldo
assigned to @oswaldo
- Author Contributor
Regarding this issue (N+1
find_commit
on discussions tab).It seems there's a lot going on on Gitaly when calling
Discussion#truncated_diff_lines
:11:01:02 gitaly.1 | time="2018-04-27T11:01:02-03:00" level=info msg="finished unary call" grpc.code=OK grpc.meta.call_site=repository_exists grpc.meta.client_name=gitlab-web grpc.method=RepositoryExists grpc.request.glRepository=project-9 grpc.request.repoPath=oswaldo/gitlab-ce.git grpc.request.repoStorage=default grpc.request.topLevelGroup=oswaldo grpc.service=gitaly.RepositoryService grpc.time_ms=0 peer.address= span.kind=server system=grpc 11:01:02 gitaly.1 | time="2018-04-27T11:01:02-03:00" level=info msg=spawn args="[/usr/local/bin/git --git-dir /Users/osw/projects/gitlab/gitlab-development-kit/repositories/oswaldo/gitlab-ce.git for-each-ref --count=1 refs/heads]" grpc.meta.call_site=has_local_branches grpc.meta.client_name=gitlab-web grpc.method=HasLocalBranches grpc.request.glRepository=project-9 grpc.request.repoPath=oswaldo/gitlab-ce.git grpc.request.repoStorage=default grpc.request.topLevelGroup=oswaldo grpc.service=gitaly.RepositoryService path=/usr/local/bin/git peer.address= pid=41709 span.kind=server system=grpc 11:01:02 gitaly.1 | time="2018-04-27T11:01:02-03:00" level=info msg="spawn complete" args="[/usr/local/bin/git --git-dir /Users/osw/projects/gitlab/gitlab-development-kit/repositories/oswaldo/gitlab-ce.git for-each-ref --count=1 refs/heads]" command.exitCode=0 command.inblock=0 command.maxrss=1589248 command.oublock=0 command.real_time_ms=16.088081 command.system_time_ms=3.823 command.user_time_ms=2.649 grpc.meta.call_site=has_local_branches grpc.meta.client_name=gitlab-web grpc.method=HasLocalBranches grpc.request.glRepository=project-9 grpc.request.repoPath=oswaldo/gitlab-ce.git grpc.request.repoStorage=default grpc.request.topLevelGroup=oswaldo grpc.service=gitaly.RepositoryService path=/usr/local/bin/git peer.address= pid=41709 span.kind=server system=grpc 11:01:02 gitaly.1 | time="2018-04-27T11:01:02-03:00" level=info msg="finished unary call" grpc.code=OK grpc.meta.call_site=has_local_branches grpc.meta.client_name=gitlab-web grpc.method=HasLocalBranches grpc.request.glRepository=project-9 grpc.request.repoPath=oswaldo/gitlab-ce.git grpc.request.repoStorage=default grpc.request.topLevelGroup=oswaldo grpc.service=gitaly.RepositoryService grpc.time_ms=16 peer.address= span.kind=server system=grpc 11:01:02 gitaly.1 | time="2018-04-27T11:01:02-03:00" level=info msg=spawn args="[/usr/local/bin/git --git-dir /Users/osw/projects/gitlab/gitlab-development-kit/repositories/oswaldo/gitlab-ce.git log -z --pretty=format:%H%x1f%an%x1f%ae%x1f%aI%x1f%cn%x1f%ce%x1f%cI%x1f%P --max-count=1 5091e67a5647bcd42f9a6aba30fb2e6123587da4 --]" grpc.meta.call_site=find_commit grpc.meta.client_name=gitlab-web grpc.method=FindCommit grpc.request.glRepository=project-9 grpc.request.repoPath=oswaldo/gitlab-ce.git grpc.request.repoStorage=default grpc.request.topLevelGroup=oswaldo grpc.service=gitaly.CommitService path=/usr/local/bin/git peer.address= pid=41710 span.kind=server system=grpc 11:01:02 gitaly.1 | time="2018-04-27T11:01:02-03:00" level=info msg=spawn args="[/usr/local/bin/git --git-dir /Users/osw/projects/gitlab/gitlab-development-kit/repositories/oswaldo/gitlab-ce.git cat-file --batch]" grpc.meta.call_site=find_commit grpc.meta.client_name=gitlab-web grpc.method=FindCommit grpc.request.glRepository=project-9 grpc.request.repoPath=oswaldo/gitlab-ce.git grpc.request.repoStorage=default grpc.request.topLevelGroup=oswaldo grpc.service=gitaly.CommitService path=/usr/local/bin/git peer.address= pid=41711 span.kind=server system=grpc 11:01:02 gitaly.1 | time="2018-04-27T11:01:02-03:00" level=info msg=spawn args="[/usr/local/bin/git --git-dir /Users/osw/projects/gitlab/gitlab-development-kit/repositories/oswaldo/gitlab-ce.git cat-file --batch-check]" grpc.meta.call_site=find_commit grpc.meta.client_name=gitlab-web grpc.method=FindCommit grpc.request.glRepository=project-9 grpc.request.repoPath=oswaldo/gitlab-ce.git grpc.request.repoStorage=default grpc.request.topLevelGroup=oswaldo grpc.service=gitaly.CommitService path=/usr/local/bin/git peer.address= pid=41712 span.kind=server system=grpc 11:01:02 gitaly.1 | time="2018-04-27T11:01:02-03:00" level=info msg="finished unary call" grpc.code=OK grpc.meta.call_site=find_commit grpc.meta.client_name=gitlab-web grpc.method=FindCommit grpc.request.glRepository=project-9 grpc.request.repoPath=oswaldo/gitlab-ce.git grpc.request.repoStorage=default grpc.request.topLevelGroup=oswaldo grpc.service=gitaly.CommitService grpc.time_ms=13 peer.address= span.kind=server system=grpc 11:01:02 gitaly.1 | time="2018-04-27T11:01:02-03:00" level=info msg="spawn complete" args="[/usr/local/bin/git --git-dir /Users/osw/projects/gitlab/gitlab-development-kit/repositories/oswaldo/gitlab-ce.git log -z --pretty=format:%H%x1f%an%x1f%ae%x1f%aI%x1f%cn%x1f%ce%x1f%cI%x1f%P --max-count=1 5091e67a5647bcd42f9a6aba30fb2e6123587da4 --]" command.exitCode=0 command.inblock=0 command.maxrss=1753088 command.oublock=0 command.real_time_ms=13.800416 command.system_time_ms=2.876 command.user_time_ms=2.099 grpc.meta.call_site=find_commit grpc.meta.client_name=gitlab-web grpc.method=FindCommit grpc.request.glRepository=project-9 grpc.request.repoPath=oswaldo/gitlab-ce.git grpc.request.repoStorage=default grpc.request.topLevelGroup=oswaldo grpc.service=gitaly.CommitService path=/usr/local/bin/git peer.address= pid=41710 span.kind=server system=grpc 11:01:02 gitaly.1 | time="2018-04-27T11:01:02-03:00" level=info msg="spawn complete" args="[/usr/local/bin/git --git-dir /Users/osw/projects/gitlab/gitlab-development-kit/repositories/oswaldo/gitlab-ce.git cat-file --batch]" command.exitCode=-1 command.inblock=0 command.maxrss=1441792 command.oublock=0 command.real_time_ms=10.936072999999999 command.system_time_ms=1.5070000000000001 command.user_time_ms=1.77 grpc.meta.call_site=find_commit grpc.meta.client_name=gitlab-web grpc.method=FindCommit grpc.request.glRepository=project-9 grpc.request.repoPath=oswaldo/gitlab-ce.git grpc.request.repoStorage=default grpc.request.topLevelGroup=oswaldo grpc.service=gitaly.CommitService path=/usr/local/bin/git peer.address= pid=41711 span.kind=server system=grpc 11:01:02 gitaly.1 | time="2018-04-27T11:01:02-03:00" level=info msg="spawn complete" args="[/usr/local/bin/git --git-dir /Users/osw/projects/gitlab/gitlab-development-kit/repositories/oswaldo/gitlab-ce.git cat-file --batch-check]" command.exitCode=0 command.inblock=0 command.maxrss=1421312 command.oublock=0 command.real_time_ms=8.074437999999999 command.system_time_ms=2.576 command.user_time_ms=1.684 grpc.meta.call_site=find_commit grpc.meta.client_name=gitlab-web grpc.method=FindCommit grpc.request.glRepository=project-9 grpc.request.repoPath=oswaldo/gitlab-ce.git grpc.request.repoStorage=default grpc.request.topLevelGroup=oswaldo grpc.service=gitaly.CommitService path=/usr/local/bin/git peer.address= pid=41712 span.kind=server system=grpc 11:01:02 gitaly.1 | time="2018-04-27T11:01:02-03:00" level=info msg=spawn args="[/usr/local/bin/git --git-dir /Users/osw/projects/gitlab/gitlab-development-kit/repositories/oswaldo/gitlab-ce.git log -z --pretty=format:%H%x1f%an%x1f%ae%x1f%aI%x1f%cn%x1f%ce%x1f%cI%x1f%P --max-count=1 bd48abd59d3fe6d623100ba20d91774280df3b12 --]" grpc.meta.call_site=find_commit grpc.meta.client_name=gitlab-web grpc.method=FindCommit grpc.request.glRepository=project-9 grpc.request.repoPath=oswaldo/gitlab-ce.git grpc.request.repoStorage=default grpc.request.topLevelGroup=oswaldo grpc.service=gitaly.CommitService path=/usr/local/bin/git peer.address= pid=41713 span.kind=server system=grpc 11:01:02 gitaly.1 | time="2018-04-27T11:01:02-03:00" level=info msg=spawn args="[/usr/local/bin/git --git-dir /Users/osw/projects/gitlab/gitlab-development-kit/repositories/oswaldo/gitlab-ce.git cat-file --batch]" grpc.meta.call_site=find_commit grpc.meta.client_name=gitlab-web grpc.method=FindCommit grpc.request.glRepository=project-9 grpc.request.repoPath=oswaldo/gitlab-ce.git grpc.request.repoStorage=default grpc.request.topLevelGroup=oswaldo grpc.service=gitaly.CommitService path=/usr/local/bin/git peer.address= pid=41714 span.kind=server system=grpc 11:01:02 gitaly.1 | time="2018-04-27T11:01:02-03:00" level=info msg=spawn args="[/usr/local/bin/git --git-dir /Users/osw/projects/gitlab/gitlab-development-kit/repositories/oswaldo/gitlab-ce.git cat-file --batch-check]" grpc.meta.call_site=find_commit grpc.meta.client_name=gitlab-web grpc.method=FindCommit grpc.request.glRepository=project-9 grpc.request.repoPath=oswaldo/gitlab-ce.git grpc.request.repoStorage=default grpc.request.topLevelGroup=oswaldo grpc.service=gitaly.CommitService path=/usr/local/bin/git peer.address= pid=41715 span.kind=server system=grpc 11:01:02 gitaly.1 | time="2018-04-27T11:01:02-03:00" level=info msg="finished unary call" grpc.code=OK grpc.meta.call_site=find_commit grpc.meta.client_name=gitlab-web grpc.method=FindCommit grpc.request.glRepository=project-9 grpc.request.repoPath=oswaldo/gitlab-ce.git grpc.request.repoStorage=default grpc.request.topLevelGroup=oswaldo grpc.service=gitaly.CommitService grpc.time_ms=14 peer.address= span.kind=server system=grpc 11:01:02 gitaly.1 | time="2018-04-27T11:01:02-03:00" level=info msg="spawn complete" args="[/usr/local/bin/git --git-dir /Users/osw/projects/gitlab/gitlab-development-kit/repositories/oswaldo/gitlab-ce.git log -z --pretty=format:%H%x1f%an%x1f%ae%x1f%aI%x1f%cn%x1f%ce%x1f%cI%x1f%P --max-count=1 bd48abd59d3fe6d623100ba20d91774280df3b12 --]" command.exitCode=0 command.inblock=0 command.maxrss=1765376 command.oublock=0 command.real_time_ms=14.131770999999999 command.system_time_ms=2.9290000000000003 command.user_time_ms=2.094 grpc.meta.call_site=find_commit grpc.meta.client_name=gitlab-web grpc.method=FindCommit grpc.request.glRepository=project-9 grpc.request.repoPath=oswaldo/gitlab-ce.git grpc.request.repoStorage=default grpc.request.topLevelGroup=oswaldo grpc.service=gitaly.CommitService path=/usr/local/bin/git peer.address= pid=41713 span.kind=server system=grpc 11:01:02 gitaly.1 | time="2018-04-27T11:01:02-03:00" level=info msg="spawn complete" args="[/usr/local/bin/git --git-dir /Users/osw/projects/gitlab/gitlab-development-kit/repositories/oswaldo/gitlab-ce.git cat-file --batch-check]" command.exitCode=0 command.inblock=0 command.maxrss=1425408 command.oublock=0 command.real_time_ms=8.136617000000001 command.system_time_ms=3.321 command.user_time_ms=1.692 grpc.meta.call_site=find_commit grpc.meta.client_name=gitlab-web grpc.method=FindCommit grpc.request.glRepository=project-9 grpc.request.repoPath=oswaldo/gitlab-ce.git grpc.request.repoStorage=default grpc.request.topLevelGroup=oswaldo grpc.service=gitaly.CommitService path=/usr/local/bin/git peer.address= pid=41715 span.kind=server system=grpc 11:01:02 gitaly.1 | time="2018-04-27T11:01:02-03:00" level=info msg="spawn complete" args="[/usr/local/bin/git --git-dir /Users/osw/projects/gitlab/gitlab-development-kit/repositories/oswaldo/gitlab-ce.git cat-file --batch]" command.exitCode=-1 command.inblock=0 command.maxrss=1445888 command.oublock=0 command.real_time_ms=11.246195 command.system_time_ms=1.655 command.user_time_ms=1.772 grpc.meta.call_site=find_commit grpc.meta.client_name=gitlab-web grpc.method=FindCommit grpc.request.glRepository=project-9 grpc.request.repoPath=oswaldo/gitlab-ce.git grpc.request.repoStorage=default grpc.request.topLevelGroup=oswaldo grpc.service=gitaly.CommitService path=/usr/local/bin/git peer.address= pid=41714 span.kind=server system=grpc 11:01:02 gitaly.1 | time="2018-04-27T11:01:02-03:00" level=info msg=spawn args="[/usr/local/bin/git --git-dir /Users/osw/projects/gitlab/gitlab-development-kit/repositories/oswaldo/gitlab-ce.git log -z --pretty=format:%H%x1f%an%x1f%ae%x1f%aI%x1f%cn%x1f%ce%x1f%cI%x1f%P --max-count=1 5091e67a5647bcd42f9a6aba30fb2e6123587da4 --]" grpc.meta.call_site=find_commit grpc.meta.client_name=gitlab-web grpc.method=FindCommit grpc.request.glRepository=project-9 grpc.request.repoPath=oswaldo/gitlab-ce.git grpc.request.repoStorage=default grpc.request.topLevelGroup=oswaldo grpc.service=gitaly.CommitService path=/usr/local/bin/git peer.address= pid=41716 span.kind=server system=grpc 11:01:02 gitaly.1 | time="2018-04-27T11:01:02-03:00" level=info msg=spawn args="[/usr/local/bin/git --git-dir /Users/osw/projects/gitlab/gitlab-development-kit/repositories/oswaldo/gitlab-ce.git cat-file --batch]" grpc.meta.call_site=find_commit grpc.meta.client_name=gitlab-web grpc.method=FindCommit grpc.request.glRepository=project-9 grpc.request.repoPath=oswaldo/gitlab-ce.git grpc.request.repoStorage=default grpc.request.topLevelGroup=oswaldo grpc.service=gitaly.CommitService path=/usr/local/bin/git peer.address= pid=41717 span.kind=server system=grpc 11:01:02 gitaly.1 | time="2018-04-27T11:01:02-03:00" level=info msg=spawn args="[/usr/local/bin/git --git-dir /Users/osw/projects/gitlab/gitlab-development-kit/repositories/oswaldo/gitlab-ce.git cat-file --batch-check]" grpc.meta.call_site=find_commit grpc.meta.client_name=gitlab-web grpc.method=FindCommit grpc.request.glRepository=project-9 grpc.request.repoPath=oswaldo/gitlab-ce.git grpc.request.repoStorage=default grpc.request.topLevelGroup=oswaldo grpc.service=gitaly.CommitService path=/usr/local/bin/git peer.address= pid=41718 span.kind=server system=grpc 11:01:02 gitaly.1 | time="2018-04-27T11:01:02-03:00" level=info msg="finished unary call" grpc.code=OK grpc.meta.call_site=find_commit grpc.meta.client_name=gitlab-web grpc.method=FindCommit grpc.request.glRepository=project-9 grpc.request.repoPath=oswaldo/gitlab-ce.git grpc.request.repoStorage=default grpc.request.topLevelGroup=oswaldo grpc.service=gitaly.CommitService grpc.time_ms=15 peer.address= span.kind=server system=grpc 11:01:02 gitaly.1 | time="2018-04-27T11:01:02-03:00" level=info msg="spawn complete" args="[/usr/local/bin/git --git-dir /Users/osw/projects/gitlab/gitlab-development-kit/repositories/oswaldo/gitlab-ce.git log -z --pretty=format:%H%x1f%an%x1f%ae%x1f%aI%x1f%cn%x1f%ce%x1f%cI%x1f%P --max-count=1 5091e67a5647bcd42f9a6aba30fb2e6123587da4 --]" command.exitCode=0 command.inblock=0 command.maxrss=1773568 command.oublock=0 command.real_time_ms=15.126043000000001 command.system_time_ms=3.02 command.user_time_ms=2.1220000000000003 grpc.meta.call_site=find_commit grpc.meta.client_name=gitlab-web grpc.method=FindCommit grpc.request.glRepository=project-9 grpc.request.repoPath=oswaldo/gitlab-ce.git grpc.request.repoStorage=default grpc.request.topLevelGroup=oswaldo grpc.service=gitaly.CommitService path=/usr/local/bin/git peer.address= pid=41716 span.kind=server system=grpc 11:01:02 gitaly.1 | time="2018-04-27T11:01:02-03:00" level=info msg="spawn complete" args="[/usr/local/bin/git --git-dir /Users/osw/projects/gitlab/gitlab-development-kit/repositories/oswaldo/gitlab-ce.git cat-file --batch-check]" command.exitCode=-1 command.inblock=0 command.maxrss=1449984 command.oublock=0 command.real_time_ms=8.605404 command.system_time_ms=2.676 command.user_time_ms=1.719 grpc.meta.call_site=find_commit grpc.meta.client_name=gitlab-web grpc.method=FindCommit grpc.request.glRepository=project-9 grpc.request.repoPath=oswaldo/gitlab-ce.git grpc.request.repoStorage=default grpc.request.topLevelGroup=oswaldo grpc.service=gitaly.CommitService path=/usr/local/bin/git peer.address= pid=41718 span.kind=server system=grpc 11:01:02 gitaly.1 | time="2018-04-27T11:01:02-03:00" level=info msg="spawn complete" args="[/usr/local/bin/git --git-dir /Users/osw/projects/gitlab/gitlab-development-kit/repositories/oswaldo/gitlab-ce.git cat-file --batch]" command.exitCode=-1 command.inblock=0 command.maxrss=1458176 command.oublock=0 command.real_time_ms=12.005755 command.system_time_ms=1.805 command.user_time_ms=1.8159999999999998 grpc.meta.call_site=find_commit grpc.meta.client_name=gitlab-web grpc.method=FindCommit grpc.request.glRepository=project-9 grpc.request.repoPath=oswaldo/gitlab-ce.git grpc.request.repoStorage=default grpc.request.topLevelGroup=oswaldo grpc.service=gitaly.CommitService path=/usr/local/bin/git peer.address= pid=41717 span.kind=server system=grpc 11:01:02 gitaly.1 | time="2018-04-27T11:01:02-03:00" level=info msg=spawn args="[/usr/local/bin/git --git-dir /Users/osw/projects/gitlab/gitlab-development-kit/repositories/oswaldo/gitlab-ce.git merge-base 5091e67a5647bcd42f9a6aba30fb2e6123587da4 bd48abd59d3fe6d623100ba20d91774280df3b12]" grpc.meta.call_site=merge_base grpc.meta.client_name=gitlab-web grpc.method=FindMergeBase grpc.request.glRepository=project-9 grpc.request.repoPath=oswaldo/gitlab-ce.git grpc.request.repoStorage=default grpc.request.topLevelGroup=oswaldo grpc.service=gitaly.RepositoryService path=/usr/local/bin/git peer.address= pid=41719 span.kind=server system=grpc 11:01:02 gitaly.1 | time="2018-04-27T11:01:02-03:00" level=info msg="spawn complete" args="[/usr/local/bin/git --git-dir /Users/osw/projects/gitlab/gitlab-development-kit/repositories/oswaldo/gitlab-ce.git merge-base 5091e67a5647bcd42f9a6aba30fb2e6123587da4 bd48abd59d3fe6d623100ba20d91774280df3b12]" command.exitCode=0 command.inblock=0 command.maxrss=1691648 command.oublock=0 command.real_time_ms=8.281519 command.system_time_ms=3.04 command.user_time_ms=1.99 grpc.meta.call_site=merge_base grpc.meta.client_name=gitlab-web grpc.method=FindMergeBase grpc.request.glRepository=project-9 grpc.request.repoPath=oswaldo/gitlab-ce.git grpc.request.repoStorage=default grpc.request.topLevelGroup=oswaldo grpc.service=gitaly.RepositoryService path=/usr/local/bin/git peer.address= pid=41719 span.kind=server system=grpc 11:01:02 gitaly.1 | time="2018-04-27T11:01:02-03:00" level=info msg="finished unary call" grpc.code=OK grpc.meta.call_site=merge_base grpc.meta.client_name=gitlab-web grpc.method=FindMergeBase grpc.request.glRepository=project-9 grpc.request.repoPath=oswaldo/gitlab-ce.git grpc.request.repoStorage=default grpc.request.topLevelGroup=oswaldo grpc.service=gitaly.RepositoryService grpc.time_ms=8 peer.address= span.kind=server system=grpc 11:01:02 gitaly.1 | time="2018-04-27T11:01:02-03:00" level=info msg=spawn args="[/usr/local/bin/git --git-dir /Users/osw/projects/gitlab/gitlab-development-kit/repositories/oswaldo/gitlab-ce.git diff --patch --raw --abbrev=40 --full-index --find-renames ec5d53cca2df63dc948bdd7e885d2becef7c076c 5091e67a5647bcd42f9a6aba30fb2e6123587da4 -- app/assets/javascripts/boards/components/board_blank_state.js.es6]" grpc.meta.call_site=diff_between grpc.meta.client_name=gitlab-web grpc.method=CommitDiff grpc.request.glRepository=project-9 grpc.request.repoPath=oswaldo/gitlab-ce.git grpc.request.repoStorage=default grpc.request.topLevelGroup=oswaldo grpc.service=gitaly.DiffService path=/usr/local/bin/git peer.address= pid=41720 span.kind=server system=grpc 11:01:02 gitaly.1 | time="2018-04-27T11:01:02-03:00" level=info msg="spawn complete" args="[/usr/local/bin/git --git-dir /Users/osw/projects/gitlab/gitlab-development-kit/repositories/oswaldo/gitlab-ce.git diff --patch --raw --abbrev=40 --full-index --find-renames ec5d53cca2df63dc948bdd7e885d2becef7c076c 5091e67a5647bcd42f9a6aba30fb2e6123587da4 -- app/assets/javascripts/boards/components/board_blank_state.js.es6]" command.exitCode=0 command.inblock=0 command.maxrss=2088960 command.oublock=0 command.real_time_ms=12.383722 command.system_time_ms=6.766 command.user_time_ms=2.579 grpc.meta.call_site=diff_between grpc.meta.client_name=gitlab-web grpc.method=CommitDiff grpc.request.glRepository=project-9 grpc.request.repoPath=oswaldo/gitlab-ce.git grpc.request.repoStorage=default grpc.request.topLevelGroup=oswaldo grpc.service=gitaly.DiffService path=/usr/local/bin/git peer.address= pid=41720 span.kind=server system=grpc 11:01:02 gitaly.1 | time="2018-04-27T11:01:02-03:00" level=info msg="finished streaming call" grpc.code=OK grpc.meta.call_site=diff_between grpc.meta.client_name=gitlab-web grpc.method=CommitDiff grpc.request.glRepository=project-9 grpc.request.repoPath=oswaldo/gitlab-ce.git grpc.request.repoStorage=default grpc.request.topLevelGroup=oswaldo grpc.service=gitaly.DiffService grpc.time_ms=12 peer.address= span.kind=server system=grpc
This is somewhat related to my current investigation on https://gitlab.com/gitlab-org/gitlab-ce/issues/44052
- Oswaldo Ferreira marked this issue as related to #44052 (closed)
marked this issue as related to #44052 (closed)
- Author Contributor
I've made a few local changes in order to use the diff persisted on PG while working on #44052 (closed) (instead repetitively requesting Gitaly for data on each discussion). This will just work for discussions for the "current version" of the MR, but that will lead to way less RPC requests to Gitaly (
find_commit
,diff_between
and probably more).The change currently breaks a few behaviors when rendering the discussions, so the code still need a few tweaks. Will send a MR soon (which will address this and #44052 (closed)).
- Contributor
@oswaldo Thats great news! Did you verify this, and how? As to me it seems that caching helps, but leaves the original problem there? So we'll still have storms of requests, right?
- Author Contributor
Did you verify this, and how?
https://gitlab.com/gitlab-org/gitlab-ce/issues/45190#note_70711428 refers to the problem encountered on this issue (how to reproduce on the description); it's mainly coming from
Discussion#truncated_diff_lines
. I still don't have an answer to the/merge_requests/<iid>.json
endpoint though.So we'll still have storms of requests, right?
We'll be getting rid of lots of
find_commit
andlist_blobs_by_sha_path
requests, as we won't need this information to build "current version" discussion diffs. - Author Contributor
Although https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/18657 led to a few improvements we still experience N+1's on both
find_commit
andlist_blobs_by_sha_path
when loading the merge request page.The
find_commit
one is mainly coming fromCompareService
down toGitlab::Git::Compare
. I'll follow-up on it tomorrow / this week. - Oswaldo Ferreira mentioned in merge request !18887 (closed)
mentioned in merge request !18887 (closed)
- Rémy Coutable added priority2 severity2 and removed ~2183694 labels
- Oswaldo Ferreira mentioned in merge request !18991 (merged)
mentioned in merge request !18991 (merged)
- Author Contributor
I've sent a WIP MR that will solve it and simplify the gathering of the diff data for discussions. Some side discussion also happened on https://gitlab.com/gitlab-org/gitlab-ce/issues/37639#note_72541708.
- Author Contributor
Added a few improvement statistics on https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/18991. This seems promising.
- Douwe Maan mentioned in commit 0a43b25c
mentioned in commit 0a43b25c
- Douwe Maan closed via merge request !18991 (merged)
closed via merge request !18991 (merged)
- Stan Hu mentioned in issue #60424 (closed)
mentioned in issue #60424 (closed)
- Zeger-Jan van de Weg added devopscreate groupgitaly labels and removed 1 deleted label
added devopscreate groupgitaly labels and removed 1 deleted label