Skip to content

Materialize the same Directory digest under multiple paths

Neill Whillans requested to merge neill/cas_downloader into master

Before raising this MR, consider whether the following are required, and complete if so:

  • Unit tests
  • Metrics
  • Documentation update(s)

If not required, please explain in brief why not.

Description

This request aims to solve the issue described in #396 (closed) where the CAS downloader was failing to materialize the same Directory digest under multiple paths.

Changes proposed in this merge request:

  • Make the pending_directory_paths dictionary store a list of directory paths associated with the Directory digest hash
  • Loop over the directory path list for the Directory digest hash when populating the directories.

Validation

Spin up a local default buildgrid server (tox -e server) and create a test directory structure like:

testdir
   |
    - dir_1
   |    |
   |     - file1
   |
    - dir_2
        |
         - file1

Then use the command bgd cas upload-dir testdir to populate the CAS, which results in something like this:

casup run-test: commands[0] | bgd cas upload-dir testdir
Queueing path=[testdir/dir_2/file1]
Queueing path=[testdir/dir_2]
Queueing path=[testdir/dir_1/file1]
Queueing path=[testdir/dir_1]
Queueing path=[testdir]
Success: Pushed path=[testdir/dir_2/file1] with digest=[f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2/5]
Success: Pushed path=[testdir/dir_2] with digest=[46bd6286a8ef714b14913559d147978e28f42c16ec1311bccc0f056513a2ca28/79]
Success: Pushed path=[testdir/dir_1/file1] with digest=[f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2/5]
Success: Pushed path=[testdir/dir_1] with digest=[46bd6286a8ef714b14913559d147978e28f42c16ec1311bccc0f056513a2ca28/79]
Success: Pushed path=[testdir] with digest=[3793b59610b14ba7544e0ab1792fecef8877669d283f1a46fba51dd735cc99f8/158]

Next use the command bgd cas download-dir 3793b59610b14ba7544e0ab1792fecef8877669d283f1a46fba51dd735cc99f8/158 testdown to download the stored directory to our testdown directory.

Once complete, check the contents of the directory. You should notice that dir_1 is empty while dir_2 contains file1.

Patch buildgrid with the request changes and re-run the test, and afterwards both subdirectories contain file1

Issues addressed

This request resolves #396 (closed)

Merge request reports