Update creation of Direct Transfer batch exports to use `in_batches`
What does this MR do and why?
This MR updates BulkImports::BatchedRelationExportService#enqueue_batch_exports
to use in_batches
instead of each_batch
when iterating through a relation and creating batch export records + enqueueing the export workers. Main reason being each_batch
raising db statement timeout error when doing the first select_all
request.
[10] pry(main)> project.merge_requests.each_batch(of: 1000) { nil }
MergeRequest Load (18.6ms) SELECT "merge_requests"."id" FROM "merge_requests" WHERE "merge_requests"."target_project_id" = 838 ORDER BY "merge_requests"."id" ASC LIMIT 1 /*application:console,db_config_name:main,console_hostname:gk-m1.local,console_username:georgekoltsov,line:/app/models/concerns/each_batch.rb:62:in `each_batch'*/
MergeRequest Load (0.5ms) SELECT "merge_requests"."id" FROM "merge_requests" WHERE "merge_requests"."target_project_id" = 838 AND "merge_requests"."id" >= 89027 ORDER BY "merge_requests"."id" ASC LIMIT 1 OFFSET 1000 /*application:console,db_config_name:main,console_hostname:gk-m1.local,console_username:georgekoltsov,line:/app/models/concerns/each_batch.rb:81:in `block in each_batch'*/
The SELECT "merge_requests"."id" FROM "merge_requests" WHERE "merge_requests"."target_project_id" = 838 ORDER BY "merge_requests"."id" ASC LIMIT 1
is the one timing out as seen in Kibana.
Reducing batch size doesn't help because the initial select_all
sql is always executed regardless of the size of the batch.
One potential solution would be to pass column: :target_project_id
for the merge requests, but this method is used for all of the exported relations (issues/epics/pipelines/etc), so it would be more complex change in order to support that. in_batches
is very performant based on my tests and supports all of the exported relations. So we go with this approach for now.
Update creation of Direct Transfer batch exports to use in_batches
Changelog: fixed
Screenshots or screen recordings
Screenshots are required for UI changes, and strongly recommended for all other merge requests.
Before | After |
---|---|
How to set up and validate locally
Numbered steps to set up and validate the change are strongly suggested.
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.