Skip to content

PCWS

David Vorick requested to merge pcws into master

NOTE

BEFORE MERGING verify whether the following is true: !4851 (comment 424570173)

Overview

Create a new object called the 'pcws' or ProjectChunkWorkerSet. The pcws is a Skynet object which spins up a series of HasSector programs (one per worker) and queries the network to figure out which hosts possess the sectors for a particular chunk.

For chunks of any erasure coding with less than 40 pieces, this is equally bandwidth efficient (assuming no frame packing) to performing a HasSector lookup on a single sector. If frame packing is in play, these jobs may be more expensive.

The purpose of splitting the lookup in to a separate object is both so that we can cache the results, and also so that we can do predictive lookups if we believe that a particular chunk is more likely to be requested in the future. A simple example of a predictive lookup is on a stream download. If a user opens a download stream on a file, we can predict that the user will likely want the whole file, or may want to seek around within the file. Instead of just creating the pcws for the chunk that the user requested, we can create a pcws for every chunk in the file (or at least a significant fraction of them). This reduces seek times.

The pcws was built so that it can be extended later to support traditional Sia downloads. Basically instead of populating the data structure with a series of HasSector calls, you populate the data structure with the roots established in the siafile.

side note: Right now the renter health and repair mechanism doesn't have a good way to check if a host still has a particular sector. For example, a host with a failed drive will lose some sectors and the renter may not have a good way to figure out which sectors of a siafile are still available. We can potentially use the PCWS in the health loop (future MR) to compare the host responses from a HasSector query to the list of sectors that are supposed to be available. An alternative check would be needed to ensure that the sector appears in the file contract.

Checklist

Review and complete the checklist to ensure that the MR is complete before assigned to an approver.

  • All new methods or updated methods have clear docstrings
  • Testing added or updated for new methods
  • Any new packages are added to Makefile and .gitlab-ci.yml
  • API documentation updated for API updates
  • Module README.md updated for changes to workflow
  • Issue added to Sia-UI repo for new supporting features
  • Changelog File Created
Edited by Peter-Jan Brone

Merge request reports