Problem with git lfs mounts in execution worker when Retrieve operators run in parallel
Summary
Pipeline fails with lfs-error when multiple repository entries are retrieved.
Actual behavior
Running a pipeline with 2 retrieve-operators without preconditions (so they can start in parallel) that both need to checkout repositories/ fetch lfs files results (~90% of the time) in following error for one of the entries:
Mon Oct 07 11:22:07 UTC 2024 Caused by: java.io.IOException: Illegal data format for entry
Mon Oct 07 11:22:07 UTC 2024 at com.owc.singularity.repository.entry.Entry$EntryHeader.<init>(Entry.java:64) ~[repository-0.9.9.jar:0.9.9]
Mon Oct 07 11:22:07 UTC 2024 at com.owc.singularity.repository.entry.Entry.getHeader(Entry.java:322) ~[repository-0.9.9.jar:0.9.9]
Mon Oct 07 11:22:07 UTC 2024 at com.owc.singularity.repository.entry.Entry.newDataInputStream(Entry.java:189) ~[repository-0.9.9.jar:0.9.9]
Mon Oct 07 11:22:07 UTC 2024 at com.owc.singularity.repository.entry.Entry.loadData(Entry.java:254) ~[repository-0.9.9.jar:0.9.9]
Mon Oct 07 11:22:07 UTC 2024 at com.owc.singularity.repository.entry.Entries.loadData(Entries.java:102) ~[repository-0.9.9.jar:0.9.9]
Mon Oct 07 11:22:07 UTC 2024 at com.owc.singularity.engine.operator.io.RetrieveOperator.read(RetrieveOperator.java:91) ~[?:?]
Mon Oct 07 11:22:07 UTC 2024 ... 9 moreExpected behavior
Parallel retrieve operations should work.