Skip to content
GitLab
Next
Projects Groups Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in / Register
  • GitLab GitLab
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
    • Locked Files
  • Issues 44,761
    • Issues 44,761
    • List
    • Boards
    • Service Desk
    • Milestones
    • Iterations
    • Requirements
  • Merge requests 1,330
    • Merge requests 1,330
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
    • Test Cases
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Packages and registries
    • Packages and registries
    • Package Registry
    • Container Registry
    • Infrastructure Registry
  • Monitor
    • Monitor
    • Metrics
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Code review
    • Insights
    • Issue
    • Repository
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • GitLab.orgGitLab.org
  • GitLabGitLab
  • Issues
  • #25252
Closed
Open
Issue created Nov 15, 2018 by James Lopez@jameslopezMaintainer

Refactor Import/Export worker model

After https://gitlab.com/gitlab-org/gitlab-ce/issues/54084 is done, we'll get more timeouts since more batching and frequent DB commits would reduce the memory footprint but slow down the imports/exports.

We should try to logically separate the Import/Export into specific workers. For this, we should consider:

  • A worker for importing the repo
  • A worker for uploads
  • Many workers for the DB

For the DB-side, we should be careful with the approach as there could be dependent models. We'll need to logically separate these workers so there's no conflict and making sure the order does not matter so we can run them concurrently... This may potentially introduce a new configuration or a spec that would detect conflicts.

A manage worker will have to coordinate the different parts and set the import to "done" when everything finishes.

We should also cope with retrying mechanisms (not currently working).

Edited Nov 15, 2018 by James Lopez
Assignee
Assign to
Time tracking