Speed up methods that depend on diff stats only
What does this MR do?
Speeds up and reduces memory usage of Commit#repo_changes
loop, Repository#expire_avatar_cache
and IrkerWorker
. It's a WIP because it need a new version of gitlab_git
released with these changes.
Are there points in the code the reviewer needs to double check?
N/A
Why was this MR needed?
For large files committed, this method ends up loading them in memory which can consume lots of memory.
What are the relevant issue numbers?
https://gitlab.com/gitlab-org/gitlab-ce/issues/19441#note_13425892
Screenshots (if relevant)
N/A
Does this MR meet the acceptance criteria?
-
CHANGELOG entry added -
Documentation created/updated -
API support added -
Tests-
Added for this feature/bug -
All builds are passing
-
-
Conform by the style guides -
Branch has no merge conflicts with master
(if you do - rebase it please) -
Squashed related commits together
Merge request reports
Activity
@ahmadsherif I also applied this change to
Repository#expire_avatar_cache
, since this also was consuming lots of RAM for the same reason. We may need to audit every use ofdiffs
. :)Added 1 commit:
- e490fd7b - Use only deltas in diffs when scanning the last commit for changes in the avatar to save memory
@stanhu Looks like you found the other spot, awesome
.@ahmadsherif Is there a reason we don't make the delta mode the default? I am also seeing a cases (e.g.
CommitController#show
) which may be loading the full patches unnecessarily.mentioned in issue #20444 (closed)
Added 29 commits:
-
e490fd7b...f46d5f58 - 26 commits from branch
master
- 3e3da221 - Speed up Commit#repo_changes
- 60545cf9 - Use only deltas in diffs when scanning the last commit for changes in the avatar to save memory
- ce693e31 - Use commit deltas when counting files in IrkerWorker
Toggle commit list-
e490fd7b...f46d5f58 - 26 commits from branch
Added 93 commits:
-
ce693e31...8c8bdb79 - 90 commits from branch
master
- 06532ad9 - Speed up Commit#repo_changes
- 666a38a8 - Use only deltas in diffs when scanning the last commit for changes in the avatar to save memory
- 6db0536c - Use commit deltas when counting files in IrkerWorker
Toggle commit list-
ce693e31...8c8bdb79 - 90 commits from branch
Added 16 commits:
-
6db0536c...e63729d9 - 13 commits from branch
master
- c0d92009 - Speed up Commit#repo_changes
- 367c0d4e - Use only deltas in diffs when scanning the last commit for changes in the avatar to save memory
- d22cdde8 - Use commit deltas when counting files in IrkerWorker
Toggle commit list-
6db0536c...e63729d9 - 13 commits from branch
Reassigned to @DouweM
This is gonna conflict with https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/5401, on which I just hit
Merge when build succeeds
!Reassigned to @ahmadsherif
Added 16 commits:
-
d22cdde8...f4282412 - 13 commits from branch
master
- a16c26c9 - Speed up Commit#repo_changes
- 6eba7188 - Use only deltas in diffs when scanning the last commit for changes in the avatar to save memory
- 08c1dd34 - Use commit deltas when counting files in IrkerWorker
Toggle commit list-
d22cdde8...f4282412 - 13 commits from branch
Reassigned to @DouweM
mentioned in commit 6a928360
mentioned in merge request gitlab-com/www-gitlab-com!2910 (merged)
Mentioned in commit ubudzisz/gitlab-ce@6a928360