Long-form Renter contracts
This is the first major step towards the new Renter architecture, in which the Renter forms long-duration, large-filesize contracts and uses them to store multiple files. This should greatly improve the scalability issues observed with the current scheme (in which a separate contract is made for each file).
Though the change sounds simple in theory, it has far-reaching implications:
- Contracts can no longer be exactly sized
- The user needs to be able to set an "allowance" that governs how much to spend on contracts
- Changing the allowance should result in new contracts, but only in certain cases
- We must gracefully handle situations where the user attempts to upload more data than their allowance can support
- The user needs to be able to set an "allowance" that governs how much to spend on contracts
- We must be stricter about who we form contracts with, since the stakes are higher
- Contracts must be fully modifiable, supporting append, edit, and delete operations
- We must store the Merkle tree of each contract; otherwise we'd have to re-hash the data of every file covered by a contract in order to revise
- To keep Merkle trees small, we must store only the "4 MB hashes"
- All files must use a 4 MB piece size
- New protocols must be created for contract negotiation
- We must store the Merkle tree of each contract; otherwise we'd have to re-hash the data of every file covered by a contract in order to revise
Here's a high-level breakdown of what I've completed thus far: (note: not all of those marked "complete" are covered in this PR! Sorry for the confusion.)
-
move hostdb code to new "contractor" package -
remove concurrency, expiration, and offline checking from renter -
modify contractor to use long-form contracts - simplest possible; uses hard-coded values
-
move HostPool to Renter -
implement set-once allowance -
add allowance API endpoints -
add metrics for how much allowance has been used -
implement auto-renewing -
renew unchanged contracts -
renew changed contracts (with new allowance) -
form new contracts if renewing fails
-
-
implement modifiable allowance -
Update on next renew cycle -
Update immediately
-
-
design and implement new negotiation protocols -
design and implement new .sia format -
achieve satisfactory test coverage -
restore concurrency and expiration -
restore multi-piece concurrency -
restore multi-chunk concurrency
-
-
delete file data after expiration - (partial; protocol not finished)
-
implement data migration (for when too much redundancy is lost) -
check for hosts going offline -
check for hosts holding data hostage -
check for hosts becoming too expensive - Note: contractor can't recover file data, only renter can
-
-
add persistence to hostdb -
contractor needs to scan blockchain to confirm that contract txns and revision txns are submitted - needs to submit them itself if not submitted by host
- can't make use of file contracts until they are seen in the blockchain
Unanswered questions:
- Should
contractor.Contracts()
return all of the contracts, or just the most recent contract formed with each host? If not the latter, how do we avoid revising outdated contracts? - What do we do if we can't form contracts with enough hosts?