Skip to content
Snippets Groups Projects

Direct Transfer - Statement timeout causes missing records

    • Open Issue created by Rodrigo Tomonari @rodrigo.tomonari

    When a database statement timeout occurs while importing records, Direct Transfer stops importing nested records, preventing them from being imported and continuing the migration.

    For example, when importing a merge request that includes multiple notes, diff notes, and events, if a statement timeout error occurs during the migration of a diff note, the merge request and its associated notes will have been migrated. Still, anything after the error will not be imported, and Direct Transfer will not attempt to retry importing them.

    Visual example:

    MergeRequest - Imported
 - Note1 - Imported
 - Note2 - Imported
 - Note3 - Imported
 - DiffNote1 <-- Error
 - DiffNote2
 - Event1
 - Event2

    MergeRequest, Note1, Note2, and Note3 are imported; everything after the error, DiffNote1, DiffNote2, Event1, and Event2 aren't.

    This problem is mainly caused by how Direct Transfer imports nested records, as explained below.

    How Direct Transfer saves nested records

    Direct Transfers save imported records from NDJSON in the load phase.

    NdjsonPipeline#load receives a data object that is an NDJSON row converted into an ActiveRecord model.

    For example, for merge_requests.ndjson, data will be a MergeRequest model with all nested associations built. Something like:

    #<MergeRequest id: root/mrs!1, 
  approvals: [#<Approval id: nil, user_id: 1], 
  notes: [
    #<Note id: nil, note: 'test', ...>, 
    #<Note id: nil, note: 'another note'>,
    #<DiffNote id: nil, note 'Diff Note'>,
    #<Note id: nil, note: 'another note'>,
  ],
  events: [
    #<Event id: nil, action: 1, user_id: 1>
  ]
  ...
>

    Then this data object is passed to Gitlab::ImportExport::Base::RelationObjectSave which saves the objects.

    Keep in mind that, theoretically, Direct Transfer could trigger data.save! since data is an ActiveRecord, allowing Rails to save everything automatically. However, because some records might contain thousands of nested records, this could result in the database transaction remaining open for an extended period. Therefore, the existing solution implemented by RelationObjectSave is to save the nested records in batches.

    Since the nested objects are saved using multiple database transactions, only the latest transaction is rolled back when an error occurs, causing NDJSON rows to be partially imported.

    Edited by Rodrigo Tomonari

    Designs

      Child items ...

      2. Activity

      Please register or sign in to reply