Skip to content

WIP: Add batching and caching to project export serialization

Aleksei Lipniagov requested to merge 35389-reduce-project-export-rss into master

What does this MR do?

Replace built-in as_json with very similar implementation but including in_batches and cache for already serialized values that are common across many entities (e.g. Milestone).
It should help to reduce peak RSS of the Project Export job.

I took methods from Rails as a foundation and tweaked them a bit.

I am considering storing only strings in 'cache' (convert to json before putting in).

Screenshots

Does this MR meet the acceptance criteria?

Conformity

Performance and Testing

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • Label as security and @ mention @gitlab-com/gl-security/appsec
  • The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • Security reports checked/validated by a reviewer from the AppSec team

Related to #35389 (moved)

Edited by 🤖 GitLab Bot 🤖

Merge request reports