Skip to content

Export via streaming serializer, introduce "Writer" abstraction

Summary

As part of our effort to introduce .ndjson as a way to process imports(&2734), we should ship streaming serializer, and refactor legacy json to use streaming serializer.

It will be an improvement on its own, since it would bring constant memory usage on exporting side for legacy format (big file), because we don't need to store the whole JSON in memory anymore as we write and forget...

We should:

  • Introduce streaming serializer as a drop-in replacement for FastHashSerializer
  • a Writer can persist relations in different ways
  • it would still produce a "fat JSON" so no ndjson here yet
  • there would be no structural changes here yet, it's mostly a refactor

Links / references

POC MR: !23920 (closed)

Epic: &2734