Skip to content
Snippets Groups Projects

catfile: Introduce request queues to allow batching reads

Merged Patrick Steinhardt requested to merge pks-catfile-queue into master
1 unresolved thread

This MR introduces request queues into the catfile package to allow batching reads. This solves the problem that we're creating too many tracing spans, and also allows us to start using batched I/O for requesting objects from commands in a follow-up MR. This MR here significantly reduces the number of allocations:

  • 17% less allocations in ListAllBlobs with contents, 40% less without contents
  • 25% less allocations in ListBlobs
  • 17% less allocations in ListAllCommits
  • 22% less allocations in FindAllTags

Also, FindAllTags is sped up by about 30% by this change.

Closes #3783 (closed)

Merge request reports

Loading
Loading

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
  • Pavlo Strokov
  • Pavlo Strokov
  • Pavlo Strokov
  • Pavlo Strokov
  • Pavlo Strokov
  • Pavlo Strokov
  • Pavlo Strokov
  • Pavlo Strokov removed review request for @8bitlife

    removed review request for @8bitlife

  • added 40 commits

    • 4e705273...c959b634 - 28 commits from branch master
    • 3b4c20bd - commit: Fix discarding of tree entries
    • 6c79e987 - blob: Extend benchmarks listing blobs
    • 1a72fa43 - catfile: Make tracing safe for concurrent use
    • f6f833f8 - catfile: Fix error messages on cached process type mismatches
    • 76696977 - catfile: Move together object struct and its receiver functions
    • 954d1672 - catfile: Move remaining bytes into object
    • 4a5ccfaa - catfile: Disentangle object reader's and object's closed status
    • 6a42efb2 - catfile: Split out object reader queue
    • 084ae943 - catfile: Split out object info reader queue
    • 94cb180a - catfile: Introduce queue interfaces for batching requests
    • 3c04de19 - gitpipe: Convert object pipeline to use object reader queue
    • 2d84bd15 - gitpipe: Convert object info pipeline to use object info queue

    Compare with previous version

  • requested review from @8bitlife

  • Pavlo Strokov approved this merge request

    approved this merge request

  • Pavlo Strokov removed review request for @8bitlife

    removed review request for @8bitlife

  • added 24 commits

    • 2d84bd15...a49570e7 - 11 commits from branch master
    • c4e8fe30 - commit: Fix discarding of tree entries
    • 5439315d - blob: Extend benchmarks listing blobs
    • 5e1194f5 - catfile: Make tracing safe for concurrent use
    • 9215390a - catfile: Drop `finish()` function returned when creating traces
    • dbf5c9d3 - catfile: Fix error messages on cached process type mismatches
    • c469c870 - catfile: Move together object struct and its receiver functions
    • 5a9107ff - catfile: Move remaining bytes into object
    • ae77398c - catfile: Disentangle object reader's and object's closed status
    • 270644ee - catfile: Split out object reader queue
    • 06a7e6d8 - catfile: Split out object info reader queue
    • fa7c6cd3 - catfile: Introduce queue interfaces for batching requests
    • 196235a3 - gitpipe: Convert object pipeline to use object reader queue
    • 73b8445f - gitpipe: Convert object info pipeline to use object info queue

    Compare with previous version

  • Sami Hiltunen approved this merge request

    approved this merge request

  • Sami Hiltunen started a merge train

    started a merge train

  • merged

  • Sami Hiltunen mentioned in commit 06ec7a17

    mentioned in commit 06ec7a17

  • added workflowcanary label and removed workflowstaging label

  • added workflowproduction label and removed workflowcanary label

  • Please register or sign in to reply
    Loading