WIP: Make project tree serialisation for export to be of constant memory
What does this MR do?
Make constant-memory export serializer
This makes the serialization to be executed lazily, using memory-optimised approach, and being fast to recycle.
The biggest benefit comes from the fact that storing full Hash of object serialization is expensive, as it also often results in holding original representations of objects.
This solves that, by making batch serialization, that outputs exact raw JSON content to generated string.
JSON generator is optimised to have efficient string appender, this makes use of that.
This brings big memory usage benefits as described in: https://gitlab.com/gitlab-org/gitlab-ce/issues/35389#note_211182737 and confirmed by two independent installations.
- Shorten export by about 4x times with N+1 fix,
- We can reduce memory footprint (for gitlabhq) from around 700MB (on top of process), to around 150MB.
Build on top of https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/32423. Per.: https://gitlab.com/gitlab-org/gitlab-ce/issues/35389#note_211182737
Does this MR meet the acceptance criteria?
Conformity
-
Changelog entry for user-facing changes, or community contribution. Check the link for other scenarios. -
Code review guidelines -
Merge request performance guidelines -
Style guides
Related to https://gitlab.com/gitlab-org/gitlab-ee/issues/26910 and https://gitlab.com/gitlab-org/gitlab-ce/issues/35389.