Skip to content

GitLab Next

  • Projects
  • Groups
  • Snippets
  • Help
    • Loading...
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in / Register
GitLab
GitLab
  • Project overview
    • Project overview
    • Details
    • Activity
    • Releases
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
    • Locked Files
  • Issues 34,942
    • Issues 34,942
    • List
    • Boards
    • Labels
    • Service Desk
    • Milestones
    • Iterations
  • Merge Requests 1,268
    • Merge Requests 1,268
  • Requirements
    • Requirements
    • List
  • CI / CD
    • CI / CD
    • Pipelines
    • Jobs
    • Schedules
    • Test Cases
  • Security & Compliance
    • Security & Compliance
    • Dependency List
    • License Compliance
  • Operations
    • Operations
    • Metrics
    • Incidents
    • Environments
  • Packages & Registries
    • Packages & Registries
    • Container Registry
  • Analytics
    • Analytics
    • CI / CD
    • Code Review
    • Insights
    • Issue
    • Repository
    • Value Stream
  • Snippets
    • Snippets
  • Members
    • Members
  • Collapse sidebar
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
  • GitLab.org
  • GitLabGitLab
  • Issues
  • #208803

Closed
Open
Opened Mar 03, 2020 by Nikola Milojevic@nmilojevic1Developer

Export via streaming serializer, introduce "Writer" abstraction

Summary

As part of our effort to introduce .ndjson as a way to process imports(&2734), we should ship streaming serializer, and refactor legacy json to use streaming serializer.

It will be an improvement on its own, since it would bring constant memory usage on exporting side for legacy format (big file), because we don't need to store the whole JSON in memory anymore as we write and forget...

We should:

  • Introduce streaming serializer as a drop-in replacement for FastHashSerializer
  • a Writer can persist relations in different ways
  • it would still produce a "fat JSON" so no ndjson here yet
  • there would be no structural changes here yet, it's mostly a refactor

Links / references

POC MR: !23920 (closed)

Epic: &2734

Assignee
Assign to
12.9
Milestone
12.9 (Past due)
Assign milestone
Time tracking
None
Due date
None
Reference: gitlab-org/gitlab#208803