Refactor Import/Export JSON load/dumping
Currently, we load the JSON in batches, freeing objects from memory per batch and committing to the DB after each batch. This is, however, not done at JSON level. We still have to load all JSON first, then we create the ActiveRecord
objects per batch and free them after.
While this has improved memory, the disadvantage is that it's not easily tweaked (possible, but the code complexity around this would grow unnecessarily).
In the export side of things, we still export everything in one go (having the advantage of faster exports) but batching should also be implemented in order to lower the memory footprint.
Note that gems such as Oj
and similar won't help much here. We probably want to roll our own version of the to_json
ActiveRecord mechanism, keeping the existent functionality. This should be a refactor per se, so the configuration would remain the same and old imports should still work.