Skip to content

Renter pools

David Vorick requested to merge renter-pools into master

None of the tests are passing, this PR is being opened mostly for a high-level pass on what's happening.

Basically, there's a worker pool now in the renter. Each worker has a contract and can perform uploads and downloads with the host on the other side of the contract.

The download queue will download 1 file at a time, with a moderate amount of parallelism. There is not really much randomization happening, which might mean it's somewhat inferior to the previous download scheme. However, I believe that the previous download scheme was entirely serial, whereas this download scheme will download an entire chunk in parallel, and it'll also download multiple chunks at the same time.

I would like to expand it so that it can also download multiple files at a time, but the code needs more work before getting there.

The uploading has much stronger parallelization now. The previous scheme was bottlenecked by the slowest host, basically meaning that uploads could move at-best as fast as 24x the slowest host, though I think in practice long timeouts meant that things usually moved very slowly, especially when trying to get to the first redundancy.

Things are still technically bottlenecked by the slowest host, however faster hosts will move onto the next chunk instead of waiting for the slower hosts. The time to reach 1x or 2x redundancy should very dramatically reduced.

What happens right now is a giant matrix of work is created when doing an upload. Every chunk that might need attention is looked at, and there's a grid that sort of remembers how much redundancy each chunk has already. 4 workers are set upon each chunk, meaning the chunk gains 1x redundancy at a time. When parallelizing, the algorithm waits until there are enough workers available and then it scans through and finds a chunk that is missing pieces from at least 4 of the available workers. Then it performs those uploads.

There's still a ton of room for optimization, and the giant upload matrices don't scale very well. But things should be moving a lot faster now vs. earlier. Once the bugs are worked out, at least.

Merge request reports