Skip to content

Make constant-memory export serializer

What does this MR do?

Make constant-memory export serializer

This makes the serialization to be executed lazily, using memory-optimised approach, and being fast to recycle.

The biggest benefit comes from the fact that storing full Hash of object serialization is expensive, as it also often results in holding original representations of objects.

This solves that, by making batch serialization, that outputs exact raw JSON content to generated string.

JSON generator is optimised to have efficient string appender, this makes use of that.

This brings big memory usage benefits as described in: https://gitlab.com/gitlab-org/gitlab-ce/issues/35389#note_211182737 and confirmed by two independent installations.

  1. Shorten export by about 4x times with N+1 fix,
  2. We can reduce memory footprint (for gitlabhq) from around 700MB (on top of process), to around 150MB.

Does this MR meet the acceptance criteria?

Conformity

Related to #26910 (closed)

Edited by 🤖 GitLab Bot 🤖

Merge request reports