Introduce support for bulk-inserting model associations
What does this MR do?
Adds support for bulk-inserting associations safely.
- References: #196844 (closed)
- Builds on: !25590 (merged)
- Feature flag:
:import_bulk_inserts
New bulk insertion API for model associations
Bulk insertions are crucial for storing large amounts of data efficiently. However, we also identified the need for this to happen in a safe manner, i.e. by ensuring bulk insertions are only available when we can have certain guarantees that we are not causing integrity problems or violate business rules (often encoded in ActiveRecord validations.)
This MR extends on !24168 (merged) and !25590 (merged) in the following ways:
BulkInsertableAssociations
: insert has_many
associations in bulk
Given a type that is BulkInsertSafe
, if it appears on the "owned" end of a relationship such as has_many
, we can now bulk-insert these records via a simple switch.
class MergeRequestDiff
include BulkInsertableAssociations
has_many :merge_request_diff_commits
end
parent = MergeRequestDiff.new
diff_commits << [...]
BulkInsertableAssociations.with_bulk_insert do
parent.save # this will insert all pending `diff_commits` in bulk
end
Internally this is realized using an after_create
hook. This way we can exploit transactionality of AR's callback chains. This API, while highly generic, has a few limitations currently:
- only
has_many
associations are supported - the
through
option is not supported - only new items can be inserted in bulk; already persisted entities, including the owner, will be ignored
For instance, this does currently not work:
parent = Parent.find_by_id ... # existing record
parent.assoc << ...
BulkInsertableAssociations.with_bulk_insert do
parent.save # this will perform a normal insert
end
The reason is that internally we rely on BulkInsertSafe.bulk_insert
from the previous MR, which in turn relies on insert_all
i.e. we can currently not perform updates/upserts. We expect to add this functionality in future iterations.
Does this MR meet the acceptance criteria?
Conformity
- [-] Changelog entry
-
Documentation (done in !26367 (merged)) -
Code review guidelines -
Merge request performance guidelines -
Style guides -
Database guides - [-] Separation of EE specific content
Availability and Testing
-
Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process. - [-] Tested in all supported browsers
- [-] Informed Infrastructure department of a default or new setting change, if applicable per definition of done
We have a great test case for this, which is project imports:
-
Run project import with bulk-inserts via associations and measure results. Repeat this test on staging once this MR is merged.