Support concurrent dependency file read, and digest calculation.

Description

Currently will spawn 4 threads for digest calculation if the number of dependency files is greater than 50. Also, allows the caller to use all available cores on the machine using a boolean flag.

Benchmark

Using a fairly large project C++ project, on a machine with 21 physical cores, was able to decrease Merkle tree building time by ~160% on average.

The raw metrics are below (Note: the concurrent build used the SKIP_CACHE option, but that doesn't affect the time it takes to build the Merkle tree. I can paste exact output, with the same options if necessary).

Non-concurrent Merkle tree build

recc.fetch_write_results:33|ms
recc.compiler_deps:1919|ms
recc.query_action_cache:237|ms
*recc.build_merkle_tree:1177|ms*
recc.find_missing_blobs:11038|ms
recc.upload_missing_blobs:791|ms
recc.execute_action:3770|ms
recc.calculate_digests_total:53|ms
recc.fetch_write_results:18|ms
recc.compiler_deps:3042|ms
recc.query_action_cache:193|ms
*recc.build_merkle_tree:1174|ms*
recc.find_missing_blobs:2958|ms
recc.upload_missing_blobs:160|ms
recc.execute_action:4495|ms
recc.calculate_digests_total:48|ms
recc.fetch_write_results:16|ms
recc.compiler_deps:1309|ms
recc.query_action_cache:230|ms
*recc.build_merkle_tree:1038|ms*
recc.find_missing_blobs:2881|ms
recc.upload_missing_blobs:176|ms
recc.execute_action:2903|ms
recc.calculate_digests_total:48|ms
recc.fetch_write_results:40|ms
recc.compiler_deps:1689|ms
recc.query_action_cache:212|ms
*recc.build_merkle_tree:1204|ms*
recc.find_missing_blobs:11251|ms
recc.upload_missing_blobs:381|ms
recc.execute_action:3101|ms
recc.calculate_digests_total:54|ms

Concurrent Merkle tree build (21 threads).

recc.fetch_write_results:42|ms
recc.execute_action:3360|ms
recc.find_missing_blobs:14531|ms
recc.compiler_deps:1441|ms
*recc.build_merkle_tree:107|ms*
recc.calculate_digests_total:75|ms
recc.fetch_write_results:14|ms
recc.execute_action:2750|ms
recc.find_missing_blobs:3121|ms
recc.compiler_deps:1224|ms
*recc.build_merkle_tree:99|ms*
recc.calculate_digests_total:53|ms
recc.fetch_write_results:28|ms
recc.execute_action:2631|ms
recc.find_missing_blobs:8728|ms
recc.compiler_deps:1447|ms
*recc.build_merkle_tree:110|ms*
recc.calculate_digests_total:71|ms
recc.fetch_write_results:31|ms
recc.execute_action:3281|ms
recc.find_missing_blobs:11389|ms
recc.compiler_deps:1502|ms
*recc.build_merkle_tree:111|ms*
recc.calculate_digests_total:69|ms

Todo

• Refactor to make testing easier.
• Add tests
• Benchmark

Issue Addressed

Closes #66 (closed)