Merge request creation times out when updating a single file due to slow 'git log --follow' performance
Summary
When creating an MR on a branch which changed only one file, FindCommits will be called with the --follow
flag. On some repos this can be very expensive, causing FindCommits
to exceed its 30 second deadline and block the MR from being created.
This is similar to #229963 in that a slow Gitaly call is blocking MR creation, but the specific call is different.
Note: The repo that triggered this problem is 16 GiB large and has an unusual usage pattern. Most files are never changed, instead when updates are required the file is copied, modified, and renamed. As a result, there are many more candidates for --follow
to search than a typical repo, as most files are slightly modified copies of each other.
git log
without the --follow
flag performs normally, and in our tests bumping the --find-renames
to 90% resolves the performance problem.
Steps to reproduce
- Create a large repo with many similar files
- Create a branch and update/add one file that closely matches the similar files
- Try to create an MR, the request times out.
What is the current bug behavior?
MR creation fails on branches which only update one file.
What is the expected correct behavior?
The MR is successfully created.
Relevant logs and/or screenshots
{
"correlation_id": "rmYr9unTkP4",
"error": "rpc error: code = Canceled desc = error streaming commits: rpc error: code = Internal desc = transport: transport: the stream is done or WriteHeader was already called",
"grpc.code": "Canceled",
"grpc.meta.auth_version": "v2",
"grpc.meta.client_name": "gitlab-web",
"grpc.meta.deadline_type": "regular",
"grpc.method": "FindCommits",
"grpc.request.deadline": "2020-06-25T12:33:33Z",
"grpc.request.fullMethod": "/gitaly.CommitService/FindCommits",
"grpc.request.glProjectPath": "group/project",
"grpc.request.glRepository": "project-1",
"grpc.request.repoPath": "group/project.git",
"grpc.request.repoStorage": "default",
"grpc.request.topLevelGroup": "group",
"grpc.service": "gitaly.CommitService",
"grpc.start_time": "2020-06-25T12:33:03Z",
"grpc.time_ms": 30022.977,
"level": "info",
"msg": "finished streaming call with code Canceled",
"peer.address": "@",
"pid": 9743,
"span.kind": "server",
"system": "grpc",
"time": "2020-06-25T12:33:33.057Z"
}
{
"args": [
"/opt/gitlab/embedded/bin/git",
"--git-dir",
"/var/opt/gitlab/git-data/repositories/group/project.git",
"log",
"--format=format:%H",
"--max-count=25",
"--follow",
"BRANCH_NAME",
"--",
"FILE_NAME"
],
"command.exitCode": -1,
"command.inblock": 0,
"command.maxrss": 793440,
"command. oublock": 0,
"command.real_time_ms": 29102.109008,
"command.system_time_ms": 612.452,
"command.user_time_ms": 28224.328,
"correlation_id": "W4uft4X5sW5",
"grpc.meta.auth_version": "v2",
"grpc.meta.client_name": "gitlab-web",
"grpc.meta.deadline_type": "regular",
"grpc.method": "FindCommits",
"grpc.request.deadline": "2020-07-21T11:08:49Z",
"grpc.request.fullMethod": "/gitaly.CommitService/FindCommits",
"grpc.request.glProjectPath": "group/project",
"grpc.request.glRepository": "project-1",
"grpc.request.repoPath": "group/project.git",
"grpc.request.repoStorage": "default",
"grpc.request.topLevelGroup": "group",
"grpc.service": "gitaly.CommitService",
"grpc.start_time": "2020-07-21T11:08:19Z",
"level": "debug",
"msg": "spawn complete",
"path": "/opt/gitlab/embedded/bin/git",
"peer.address": "@",
"pid": 9708,
"span.kind": "server",
"system": "grpc",
"time": "2020-07-21T11:08:49.06Z"
}
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: Ubuntu 18.04 Proxy: no Current User: git Using RVM: no Ruby Version: 2.6.6p146 Gem Version: 2.7.10 Bundler Version:1.17.3 Rake Version: 12.3.3 Redis Version: 5.0.9 Git Version: 2.26.2 Sidekiq Version:5.2.7 Go Version: unknown GitLab information Version: 13.0.6-ee Revision: 4c8f8b0d9ee Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 11.7 URL: REDACTED HTTP Clone URL: REDACTED SSH Clone URL: REDACTED Elasticsearch: no Geo: no Using LDAP: yes Using Omniauth: yes Omniauth Providers: GitLab Shell Version: 13.2.0 Repository storage paths: - default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 13.2.0 ? ... OK (13.2.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... Server: ldapmain LDAP authentication... Anonymous. No
bind_dn
orpassword
configured LDAP users with access to your GitLab server (only showing the first 100 results) User output sanitized. Found 100 users of 100 limit.Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 3/5 ... yes 114/13 ... yes 56/17 ... yes 56/19 ... yes 3/22 ... yes 99/26 ... yes 99/28 ... yes 2/29 ... yes 2/30 ... yes 3/34 ... yes 99/37 ... yes 99/38 ... yes 99/40 ... yes 99/42 ... yes 45/55 ... yes 2/56 ... yes 26/58 ... yes 26/59 ... yes 2/60 ... yes 42/65 ... yes 42/67 ... yes 2/68 ... yes 2/75 ... yes 2/76 ... yes 45/77 ... yes 45/78 ... yes 90/87 ... yes 2/88 ... yes 26/95 ... yes 25/96 ... yes 96/97 ... yes 96/98 ... yes 96/99 ... yes 96/101 ... yes 105/102 ... yes 56/113 ... yes 90/115 ... yes 3/121 ... yes 3/122 ... yes 3/123 ... yes 99/127 ... yes 46/129 ... yes 91/132 ... yes 91/133 ... yes 45/134 ... yes 45/135 ... yes 45/136 ... yes 45/137 ... yes 45/138 ... yes 45/139 ... yes 26/158 ... yes 35/160 ... yes 36/164 ... yes 35/166 ... yes 36/167 ... yes 45/171 ... yes 35/178 ... yes 36/180 ... yes 3/183 ... yes 42/185 ... yes 91/186 ... yes 45/187 ... yes 45/191 ... yes 56/192 ... yes 56/193 ... yes 90/194 ... yes 3/196 ... yes 45/197 ... yes 91/199 ... yes 42/200 ... yes 26/201 ... yes 26/203 ... yes 91/204 ... yes 3/205 ... yes 54/208 ... yes 76/209 ... yes 3/210 ... yes 5/211 ... yes 91/212 ... yes 59/213 ... yes 74/214 ... yes 74/215 ... yes 74/216 ... yes 26/217 ... yes 45/218 ... yes 26/221 ... yes 42/226 ... yes 59/227 ... yes 90/228 ... yes 42/232 ... yes 90/234 ... yes 91/235 ... yes 42/236 ... yes 74/238 ... yes 3/239 ... yes 45/244 ... yes 42/245 ... yes 3/247 ... yes 76/248 ... yes 96/249 ... yes 33/250 ... yes 26/252 ... yes 3/253 ... yes 59/254 ... yes 80/257 ... yes 42/259 ... yes 80/260 ... yes 99/261 ... yes 90/264 ... yes 90/265 ... yes 45/266 ... yes 75/267 ... yes 42/268 ... yes 54/270 ... yes 49/271 ... yes 99/272 ... yes 75/275 ... yes 88/276 ... yes 3/277 ... yes 26/278 ... yes 26/279 ... yes 42/280 ... yes 90/281 ... yes 3/282 ... yes 42/284 ... yes 121/286 ... yes 45/291 ... yes 75/292 ... yes 76/293 ... yes 42/294 ... yes 3/295 ... yes 45/296 ... yes 99/297 ... yes 97/300 ... yes 88/301 ... yes 26/303 ... yes 56/307 ... yes 74/309 ... yes 103/311 ... yes 3/312 ... yes 26/313 ... yes 26/314 ... yes 105/315 ... yes 101/316 ... yes 114/321 ... yes 59/329 ... yes 80/330 ... yes 75/331 ... yes 49/332 ... yes 36/335 ... yes 42/336 ... yes 106/337 ... yes 3/338 ... yes 88/339 ... yes 74/340 ... yes 106/341 ... yes 42/342 ... yes 26/343 ... yes 3/344 ... yes 49/345 ... yes 26/346 ... yes 106/347 ... yes 106/348 ... yes 106/350 ... yes 74/351 ... yes 101/352 ... yes 5/353 ... yes 42/354 ... yes 75/356 ... yes 106/357 ... yes 42/358 ... yes 88/359 ... yes 106/360 ... yes 106/361 ... yes 106/364 ... yes 106/365 ... yes 42/366 ... yes 42/367 ... yes 75/369 ... yes 106/370 ... yes 114/371 ... yes 54/372 ... yes 114/373 ... yes 107/374 ... yes 114/375 ... yes 3/376 ... yes 106/377 ... yes 2/378 ... yes 2/379 ... yes 42/380 ... yes 114/383 ... yes 5/384 ... yes 114/386 ... yes 33/387 ... yes 89/388 ... yes 80/389 ... yes 106/390 ... yes 106/391 ... yes 75/392 ... yes 117/393 ... yes 49/394 ... yes 107/395 ... yes 2/396 ... yes 33/397 ... yes 33/398 ... yes 42/399 ... yes 114/400 ... yes 120/401 ... yes 75/402 ... yes 33/403 ... yes 114/404 ... yes 121/405 ... yes 3/406 ... yes 74/407 ... yes 114/408 ... yes 26/409 ... yes 5/410 ... yes 26/411 ... yes 106/412 ... yes 114/414 ... yes 90/415 ... yes 114/417 ... yes 114/418 ... yes 3/419 ... yes 106/420 ... yes 114/421 ... yes 106/422 ... yes 90/423 ... yes 49/424 ... yes 107/425 ... yes 3/426 ... yes 33/427 ... yes Redis version >= 4.0.0? ... yes Ruby version >= 2.5.3 ? ... yes (2.6.6) Git version >= 2.22.0 ? ... yes (2.26.2) Git user has default SSH configuration? ... yes Active users: ... 35 Is authorized keys file accessible? ... yes GitLab configured to store new projects in hashed storage? ... yes All projects are in hashed storage? ... no Try fixing it: Please migrate all projects to hashed storage as legacy storage is deprecated in 13.0 and support will be removed in 14.0. For more information see: doc/administration/repository_storage_types.md Elasticsearch version 5.6 - 6.x? ... skipped (elasticsearch is disabled)
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished
Possible fixes
It's unclear to me what use cases require --follow. Assuming nothing is relying on this behavior, removing this flag will resolve the issue.