Direct Transfer - Issue while creating group

For some reason, a migration got stuck as BulkImports::PipelineWorker didn't create the group as it was supposed to.

bulk_import_entity_id: 1255928

See the logs below for the sequence of events:

Kibana logs - Internal only(Valid for 7 days)

It seems that BulkImports::PipelineWorker didn't execute the pipeline because an exclusive lease wasn't obtained. However, when that happens, we are supposed to see the log json.message: "Cannot obtain an exclusive lease. There must be another instance already in execution.", which isn't present in Kibana. See these logs as an example.

Because the group wasn't created and the BulkImports::Groups::Pipelines::GroupPipeline wasn't executed, migration got stuck in stage 0 and eventually timed out.

Proposed solution

At the moment, I'm unsure about the solution. Perhaps we could remove the try_obtain_lease, but I'm concerned that it might cause the pipeline to be executed twice, as one of the reasons for adding the lock was to prevent that.