catfile: Introduce request queues to allow batching reads
This MR introduces request queues into the catfile package to allow batching reads. This solves the problem that we're creating too many tracing spans, and also allows us to start using batched I/O for requesting objects from commands in a follow-up MR. This MR here significantly reduces the number of allocations:
- 17% less allocations in ListAllBlobs with contents, 40% less without contents
- 25% less allocations in ListBlobs
- 17% less allocations in ListAllCommits
- 22% less allocations in FindAllTags
Also, FindAllTags is sped up by about 30% by this change.
Closes #3783 (closed)
Merge request reports
Activity
changed milestone to %14.5
assigned to @pks-t
mentioned in issue #3783 (closed)
4 Warnings ⚠ This merge request is quite big (1228 lines changed), please consider splitting it into multiple merge requests. ⚠ This merge request includes more than 10 commits. Each commit should meet the following criteria: - Have a well-written commit message.
- Has all tests passing when used on its own (e.g. when using git checkout SHA).
- Can be reverted on its own without also requiring the revert of commit that came before it.
- Is small enough that it can be reviewed in isolation in under 30 minutes or so.
If this merge request contains commits that do not meet this criteria and/or contains intermediate work, please rebase these commits into a smaller number of commits or split this merge request into multiple smaller merge requests.
⚠ 73b8445f: The commit body should not contain more than 72 characters per line. For more information, take a look at our Commit message guidelines. ⚠ 196235a3: The commit body should not contain more than 72 characters per line. For more information, take a look at our Commit message guidelines. Reviewer roulette
Changes that require review have been detected! A merge request is normally reviewed by both a reviewer and a maintainer in its primary category and by a maintainer in all other categories.
To spread load more evenly across eligible reviewers, Danger has picked a candidate for each review slot. Feel free to override these selections if you think someone else would be better-suited or use the GitLab Review Workload Dashboard to find other available reviewers.
To read more on how to use the reviewer roulette, please take a look at the Engineering workflow and code review guidelines. Please consider assigning a reviewer or maintainer who is a domain expert in the area of the merge request.
Once you've decided who will review this merge request, mention them as you normally would! Danger does not automatically notify them for you.
Category Reviewer Maintainer ~ Toon Claes ( @toon
) (UTC+1, same timezone as@pks-t
)Pavlo Strokov ( @8bitlife
) (UTC+2, 1 hour ahead of@pks-t
)If needed, you can retry the
danger-review
job that generated this comment.Generated by
🚫 DangerEdited by GitalyBot- Resolved by Igor
@igorwwwwwwwwwwwwwwwwwwww FYI, this implements batched traces.
Note that this depends on !4031 (merged) for the initial refactoring to allow for batched traces.
mentioned in issue #3142 (closed)
added 10 commits
- ef0cb1d2 - catfile: Make tracing safe for concurrent use
- cfb370c8 - catfile: Fix error messages on cached process type mismatches
- d988f1f9 - catfile: Move together object struct and its receiver functions
- 871f5133 - catfile: Move remaining bytes into object
- 83723a2b - catfile: Disentangle object reader's and object's closed status
- 8c7e9b68 - catfile: Split out object reader queue
- 53cbb7c5 - catfile: Split out object info reader queue
- e7aee068 - catfile: Introduce queue interfaces for batching requests
- 8ba4939e - gitpipe: Convert object pipeline to use object reader queue
- 693c0439 - gitpipe: Convert object info pipeline to use object info queue
Toggle commit listSetting label(s) Category:Gitaly based on groupgitaly.
added Category:Gitaly label
added 28 commits
-
693c0439...d03d5db1 - 16 commits from branch
master
- 8768074a - commit: Fix discarding of tree entries
- f01f949b - blob: Extend benchmarks listing blobs
- 1139fc29 - catfile: Make tracing safe for concurrent use
- 8dba57c5 - catfile: Fix error messages on cached process type mismatches
- 34d05375 - catfile: Move together object struct and its receiver functions
- ee8b4f9a - catfile: Move remaining bytes into object
- 22cce035 - catfile: Disentangle object reader's and object's closed status
- aa318bf0 - catfile: Split out object reader queue
- 7f1fd9af - catfile: Split out object info reader queue
- a0fc48c5 - catfile: Introduce queue interfaces for batching requests
- 8fe47d69 - gitpipe: Convert object pipeline to use object reader queue
- 4e705273 - gitpipe: Convert object info pipeline to use object info queue
Toggle commit list-
693c0439...d03d5db1 - 16 commits from branch
requested review from @samihiltunen and removed review request for @toon
Reassigning given that @toon is OOO
mentioned in issue gitlab#340659 (closed)
- Resolved by Patrick Steinhardt
- Resolved by Pavlo Strokov
- Automatically resolved by Patrick Steinhardt
- Automatically resolved by Pavlo Strokov
- Automatically resolved by Pavlo Strokov
- Automatically resolved by Pavlo Strokov
- Automatically resolved by Patrick Steinhardt
- Resolved by Patrick Steinhardt
- Resolved by Pavlo Strokov
@pks-t some of my comments are not relevant as they are related to code that was removed in the next commits so feel free to just resolve them silently.
removed review request for @8bitlife
added 40 commits
-
4e705273...c959b634 - 28 commits from branch
master
- 3b4c20bd - commit: Fix discarding of tree entries
- 6c79e987 - blob: Extend benchmarks listing blobs
- 1a72fa43 - catfile: Make tracing safe for concurrent use
- f6f833f8 - catfile: Fix error messages on cached process type mismatches
- 76696977 - catfile: Move together object struct and its receiver functions
- 954d1672 - catfile: Move remaining bytes into object
- 4a5ccfaa - catfile: Disentangle object reader's and object's closed status
- 6a42efb2 - catfile: Split out object reader queue
- 084ae943 - catfile: Split out object info reader queue
- 94cb180a - catfile: Introduce queue interfaces for batching requests
- 3c04de19 - gitpipe: Convert object pipeline to use object reader queue
- 2d84bd15 - gitpipe: Convert object info pipeline to use object info queue
Toggle commit list-
4e705273...c959b634 - 28 commits from branch
requested review from @8bitlife
removed review request for @8bitlife
added 24 commits
-
2d84bd15...a49570e7 - 11 commits from branch
master
- c4e8fe30 - commit: Fix discarding of tree entries
- 5439315d - blob: Extend benchmarks listing blobs
- 5e1194f5 - catfile: Make tracing safe for concurrent use
- 9215390a - catfile: Drop `finish()` function returned when creating traces
- dbf5c9d3 - catfile: Fix error messages on cached process type mismatches
- c469c870 - catfile: Move together object struct and its receiver functions
- 5a9107ff - catfile: Move remaining bytes into object
- ae77398c - catfile: Disentangle object reader's and object's closed status
- 270644ee - catfile: Split out object reader queue
- 06a7e6d8 - catfile: Split out object info reader queue
- fa7c6cd3 - catfile: Introduce queue interfaces for batching requests
- 196235a3 - gitpipe: Convert object pipeline to use object reader queue
- 73b8445f - gitpipe: Convert object info pipeline to use object info queue
Toggle commit list-
2d84bd15...a49570e7 - 11 commits from branch
mentioned in commit 06ec7a17
added workflowstaging-ref label and removed workflowready for review label
added workflowstaging-canary label and removed workflowstaging-ref label
added workflowstaging label and removed workflowstaging-canary label
added workflowcanary label and removed workflowstaging label
added workflowproduction label and removed workflowcanary label
added releasedcandidate label
added releasedpublished label and removed releasedcandidate label