WIP: Add batching and caching to project export serialization
What does this MR do?
Replace built-in as_json
with very similar implementation but including in_batches
and cache for already serialized values that are common across many entities (e.g. Milestone).
It should help to reduce peak RSS of the Project Export job.
I took methods from Rails as a foundation and tweaked them a bit.
I am considering storing only strings in 'cache' (convert to json before putting in).
Screenshots
Does this MR meet the acceptance criteria?
Conformity
-
Changelog entry for user-facing changes, or community contribution. Check the link for other scenarios. -
Documentation created/updated or follow-up review issue created -
Code review guidelines -
Merge request performance guidelines -
Style guides -
Database guides -
Separation of EE specific content
Performance and Testing
-
Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process. -
Tested in all supported browsers
Security
If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:
-
Label as security and @ mention @gitlab-com/gl-security/appsec
-
The MR includes necessary changes to maintain consistency between UI, API, email, or other methods -
Security reports checked/validated by a reviewer from the AppSec team
Related to #35389 (moved)
Edited by 🤖 GitLab Bot 🤖