Skip to content

Introduce support for bulk-inserting model associations

Matthias Käppler requested to merge 196844-safe-bulk-insert-associations into master

What does this MR do?

Adds support for bulk-inserting associations safely.

New bulk insertion API for model associations

Bulk insertions are crucial for storing large amounts of data efficiently. However, we also identified the need for this to happen in a safe manner, i.e. by ensuring bulk insertions are only available when we can have certain guarantees that we are not causing integrity problems or violate business rules (often encoded in ActiveRecord validations.)

This MR extends on !24168 (merged) and !25590 (merged) in the following ways:

BulkInsertableAssociations: insert has_many associations in bulk

Given a type that is BulkInsertSafe, if it appears on the "owned" end of a relationship such as has_many, we can now bulk-insert these records via a simple switch.

class MergeRequestDiff
  include BulkInsertableAssociations

  has_many :merge_request_diff_commits
end

parent = MergeRequestDiff.new
diff_commits << [...]

BulkInsertableAssociations.with_bulk_insert do
  parent.save # this will insert all pending `diff_commits` in bulk
end

Internally this is realized using an after_create hook. This way we can exploit transactionality of AR's callback chains. This API, while highly generic, has a few limitations currently:

  • only has_many associations are supported
  • the through option is not supported
  • only new items can be inserted in bulk; already persisted entities, including the owner, will be ignored

For instance, this does currently not work:

parent = Parent.find_by_id ... # existing record
parent.assoc << ...
BulkInsertableAssociations.with_bulk_insert do
  parent.save # this will perform a normal insert
end

The reason is that internally we rely on BulkInsertSafe.bulk_insert from the previous MR, which in turn relies on insert_all i.e. we can currently not perform updates/upserts. We expect to add this functionality in future iterations.

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

We have a great test case for this, which is project imports:

  • Run project import with bulk-inserts via associations and measure results. Repeat this test on staging once this MR is merged.
Edited by 🤖 GitLab Bot 🤖

Merge request reports