Skip to content

ExecutionClient::downloadOutputs: detect duplicate files in duplicate directories

Zehao Chen requested to merge zchen723/download-dup-detection into master

Before raising this MR, consider whether the following are required, and complete if so:

  • Unit tests
  • Metrics
  • Documentation update(s)

If not required, please explain in brief why not.

Description

Before this fix, duplicate files in duplicate directories are not detected as each equivalent directory is visited once. In such cases, downloads will fail as No file or directory.

We modified the detection algorithm to simply walk the trees instead of treating them as DAGs to make sure duplicate directories are also considered.

Merge request reports