Skip to content
GitLab
Next
Projects Groups Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in / Register
  • GitLab GitLab
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
    • Locked Files
  • Issues 44,761
    • Issues 44,761
    • List
    • Boards
    • Service Desk
    • Milestones
    • Iterations
    • Requirements
  • Merge requests 1,330
    • Merge requests 1,330
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
    • Test Cases
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Packages and registries
    • Packages and registries
    • Package Registry
    • Container Registry
    • Infrastructure Registry
  • Monitor
    • Monitor
    • Metrics
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Code review
    • Insights
    • Issue
    • Repository
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • GitLab.orgGitLab.org
  • GitLabGitLab
  • Issues
  • #25251
Closed
Open
Issue created Nov 15, 2018 by James Lopez@jameslopezMaintainer

Refactor Import/Export JSON load/dumping

Currently, we load the JSON in batches, freeing objects from memory per batch and committing to the DB after each batch. This is, however, not done at JSON level. We still have to load all JSON first, then we create the ActiveRecord objects per batch and free them after.

While this has improved memory, the disadvantage is that it's not easily tweaked (possible, but the code complexity around this would grow unnecessarily).

In the export side of things, we still export everything in one go (having the advantage of faster exports) but batching should also be implemented in order to lower the memory footprint.

Note that gems such as Oj and similar won't help much here. We probably want to roll our own version of the to_json ActiveRecord mechanism, keeping the existent functionality. This should be a refactor per se, so the configuration would remain the same and old imports should still work.

Assignee
Assign to
Time tracking