Skip to content

Update creation of Direct Transfer batch exports to use `in_batches`

George Koltsov requested to merge georgekoltsov/relation_batch_export_in_batches into master

What does this MR do and why?

This MR updates BulkImports::BatchedRelationExportService#enqueue_batch_exports to use in_batches instead of each_batch when iterating through a relation and creating batch export records + enqueueing the export workers. Main reason being each_batch raising db statement timeout error when doing the first select_all request.

[10] pry(main)> project.merge_requests.each_batch(of: 1000) { nil }
  MergeRequest Load (18.6ms)  SELECT "merge_requests"."id" FROM "merge_requests" WHERE "merge_requests"."target_project_id" = 838 ORDER BY "merge_requests"."id" ASC LIMIT 1 /*application:console,db_config_name:main,console_hostname:gk-m1.local,console_username:georgekoltsov,line:/app/models/concerns/each_batch.rb:62:in `each_batch'*/
  MergeRequest Load (0.5ms)  SELECT "merge_requests"."id" FROM "merge_requests" WHERE "merge_requests"."target_project_id" = 838 AND "merge_requests"."id" >= 89027 ORDER BY "merge_requests"."id" ASC LIMIT 1 OFFSET 1000 /*application:console,db_config_name:main,console_hostname:gk-m1.local,console_username:georgekoltsov,line:/app/models/concerns/each_batch.rb:81:in `block in each_batch'*/

The SELECT "merge_requests"."id" FROM "merge_requests" WHERE "merge_requests"."target_project_id" = 838 ORDER BY "merge_requests"."id" ASC LIMIT 1 is the one timing out as seen in Kibana.

image

Reducing batch size doesn't help because the initial select_all sql is always executed regardless of the size of the batch.

One potential solution would be to pass column: :target_project_id for the merge requests, but this method is used for all of the exported relations (issues/epics/pipelines/etc), so it would be more complex change in order to support that. in_batches is very performant based on my tests and supports all of the exported relations. So we go with this approach for now.

#434006 (closed)

Update creation of Direct Transfer batch exports to use in_batches

Changelog: fixed

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

Before After

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by George Koltsov

Merge request reports