Create a separate sidekiq queue for project template exports
Summary
As described in this original issue, creating a new project from a group-level project template on gitlab.com sometimes fails with
The repository could not be imported.
Import timed out. Import took longer than 54000 seconds
To significantly reduce the chance of this bug happening, we should create a separate sidekiq queue just for the project template exports.
Steps to reproduce
- Create a new project from a group template.
Per this comment, the exact flow is as follows:
- New project using custom template is requested
- Blank project is then created with
project.import_state = scheduled
-- in order to show 'Import in Progress' screen on the frontend - Since this is a project from template, a new export job is queued for the template http://gitlab.com/gitlab-org/gitlab/blob/master/ee/app/services/ee/projects/gitlab_projects_import_service.rb#L13-13
- Export of a template is executed in 10 minutes
- While that was happening StuckImportJob ran, found project with import state scheduled but without a jid and marked it as failed, regardless of what timeouts we have
- After export completed, an import was queued http://gitlab.com/gitlab-org/gitlab/blob/master/ee/lib/ee/gitlab/import_export/after_export_strategies/custom_template_export_import_strategy.rb#L30-30
- Lastly, the actual import failed to start with import state was marked as failed http://gitlab.com/gitlab-org/gitlab/blob/master/app/workers/repository_import_worker.rb#L46-46
Example Project
Subgroup: https://gitlab.com/edmond-demo/gitlab-days
What is the current bug behavior?
Fails with
The repository could not be imported.
Import timed out. Import took longer than 54000 seconds
What is the expected correct behavior?
New project created based on the group template.
Output of checks
This bug happens on GitLab.com
Possible fixes
Per this comment, these are some of the steps we can take to solve this problem:
- Create a new sidekiq queue just for project template export work which ensures that regular project exports do not interfere. This does not solve a race condition, but reduces the changes of it happening
- Investigate ways of not marking such project as failed, when its state is 'scheduled' and no jid is present. Perhaps we should look at the timeout setting and skip such projects until project is older than
IMPORT_JOBS_EXPIRATION
- Somewhat related, we should see if we can make ProjectExportWorker idempotent with deduplication strategy in place, in order to remove duplicate jobs from the queue if a similar job is currently running. I can imagine a situation where a user enqueues a big project export more than once, if export is not completed within a reasonable amount of time (at least that's what I would do, if I get no progress updates after some time). Such scenario is a waste of resources which can delay other jobs.
Proposed fix
In this issue, we will implement suggestion 1. from the section above (in bold):
Create a separate sidekiq queue just for the project template exports, which would ensure that regular project exports do not impact new projects being created from a template. Note that this does not solve the race condition, but reduces the changes of it happening.