Increase the IMPORT_JOBS_EXPIRATION limit
Problem
Expanded from discussion with @georgekoltsov
IMPORT_JOBS_EXPIRATION
is set to 15 hours.
Importing large projects can take a long time. I have a test project which seemed to complete:
# gitlab-rake "gitlab:import_export:import[root,test/zd158866,02_20210601_fullsize,/var/opt/gitlab/ben/2021-05-16_13.10-bigprojectv1_export.tar.gz"]
**************************************************
⛔ WARNING: Sidekiq testing API enabled, but this is not the test environment. Your jobs will not go to Redis.
**************************************************
I, [2021-06-01T10:39:25.125329 #30748] INFO -- : Importing GitLab export: /var/opt/gitlab/ben/2021-05-16_13.10-bigprojectv1_export.tar.gz into GitLab test/zd158866/02_20210601_fullsize as Administrator
I, [2021-06-02T06:35:55.155506 #30748] INFO -- : Done!
- The Repo, MRs, issues, labels, comments imported
- Pipeline import was only partial - 14,000 of the 28,000 pipelines
It actually ran for 20 hours, but when I checked in the API, confirmed that it had got reaped by IMPORT_JOBS_EXPIRATION
"import_error": "Import timed out. Import took longer than 54000 seconds"
I'm trying to reproduce issues the customer is experiencing (GitLab team members can read more in the ticket) or is likely to experience. While their MR and Issue count is lower, they have 68,000 pipelines to export and import, and their jobs record count is 100x mine.
The fix (!63065 (merged)) for #332313 (closed) will make imports more reliable, but I still suspect there's still some intermittent issues - I've had it fail several times with deadline exceeded
.
Proposal
Support a larger (much larger) IMPORT_JOBS_EXPIRATION
value, and ensure the time threshold LAST_MODIFIED_TIME_IN_MINUTES
in ImportExportCleanUpService
increases in lock-step.
NDJSON reduced the memory demands of imports, and !63065 (merged) removes the accidental deletion of the exported data before the import could complete.
Potentially for large projects it's now just a case of waiting. Inserting lots of database records takes time.
Our docs suggest using the rake task for large imports and then it's not in Sidekiq, it's a separate process.
What is the problem that IMPORT_JOBS_EXPIRATION
solves?