Skip to content

GitLab Next

  • Projects
  • Groups
  • Snippets
  • Help
    • Loading...
  • Help
    • See what's new at GitLab
    • Help
    • Support
    • Submit feedback
    • Contribute to GitLab
  • Sign in / Register
GitLab FOSS
GitLab FOSS
  • Project overview
    • Project overview
    • Details
    • Activity
    • Releases
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
    • Locked Files
  • Issues 0
    • Issues 0
    • List
    • Boards
    • Labels
    • Service Desk
    • Milestones
  • Merge Requests 0
    • Merge Requests 0
  • Requirements
    • Requirements
    • List
  • Security & Compliance
    • Security & Compliance
    • Dependency List
    • License Compliance
  • Packages & Registries
    • Packages & Registries
    • Container Registry
  • Analytics
    • Analytics
    • Code Review
    • Insights
    • Issues
    • Repository
    • Value Stream
  • Snippets
    • Snippets
  • Members
    • Members
  • Collapse sidebar
  • Activity
  • Graph
  • Create a new issue
  • Commits
  • Issue Boards
  • GitLab.org
  • GitLab FOSSGitLab FOSS
  • Issues
  • #54085

Closed (moved)
Open
Opened Nov 15, 2018 by James Lopez@jameslopez🔴
  • Report abuse
  • New issue
Report abuse New issue

Refactor Import/Export worker model

After https://gitlab.com/gitlab-org/gitlab-ce/issues/54084 is done, we'll get more timeouts since more batching and frequent DB commits would reduce the memory footprint but slow down the imports/exports.

We should try to logically separate the Import/Export into specific workers. For this, we should consider:

  • A worker for importing the repo
  • A worker for uploads
  • Many workers for the DB

For the DB-side, we should be careful with the approach as there could be dependent models. We'll need to logically separate these workers so there's no conflict and making sure the order does not matter so we can run them concurrently... This may potentially introduce a new configuration or a spec that would detect conflicts.

A manage worker will have to coordinate the different parts and set the import to "done" when everything finishes.

We should also cope with retrying mechanisms (not currently working).

Edited Nov 15, 2018 by James Lopez

Linked issues

  • Discussion
  • Designs
Assignee
Assign to
None
Milestone
None
Assign milestone
Time tracking
None
Due date
None
0
Labels
None
Assign labels
  • View project labels
Reference: gitlab-org/gitlab-foss#54085