Refactor Import/Export JSON load/dumping
Currently, we load the JSON in batches, freeing objects from memory per batch and committing to the DB after each batch. This is, however, not done at JSON level. We still have to load all JSON first, then we create the
ActiveRecord objects per batch and free them after.
While this has improved memory, the disadvantage is that it's not easily tweaked (possible, but the code complexity around this would grow unnecessarily).
In the export side of things, we still export everything in one go (having the advantage of faster exports) but batching should also be implemented in order to lower the memory footprint.
Note that gems such as
Oj and similar won't help much here. We probably want to roll our own version of the
to_json ActiveRecord mechanism, keeping the existent functionality. This should be a refactor per se, so the configuration would remain the same and old imports should still work.