Skip to content

Long-form Renter contracts

Luke Champine requested to merge contracts into renter-host-upgrades

This is the first major step towards the new Renter architecture, in which the Renter forms long-duration, large-filesize contracts and uses them to store multiple files. This should greatly improve the scalability issues observed with the current scheme (in which a separate contract is made for each file).

Though the change sounds simple in theory, it has far-reaching implications:

  • Contracts can no longer be exactly sized
    • The user needs to be able to set an "allowance" that governs how much to spend on contracts
      • Changing the allowance should result in new contracts, but only in certain cases
    • We must gracefully handle situations where the user attempts to upload more data than their allowance can support
  • We must be stricter about who we form contracts with, since the stakes are higher
  • Contracts must be fully modifiable, supporting append, edit, and delete operations
    • We must store the Merkle tree of each contract; otherwise we'd have to re-hash the data of every file covered by a contract in order to revise
      • To keep Merkle trees small, we must store only the "4 MB hashes"
      • All files must use a 4 MB piece size
    • New protocols must be created for contract negotiation

Here's a high-level breakdown of what I've completed thus far: (note: not all of those marked "complete" are covered in this PR! Sorry for the confusion.)

  • move hostdb code to new "contractor" package
  • remove concurrency, expiration, and offline checking from renter
  • modify contractor to use long-form contracts
    • simplest possible; uses hard-coded values
  • move HostPool to Renter
  • implement set-once allowance
  • add allowance API endpoints
  • add metrics for how much allowance has been used
  • implement auto-renewing
    • renew unchanged contracts
    • renew changed contracts (with new allowance)
    • form new contracts if renewing fails
  • implement modifiable allowance
    • Update on next renew cycle
    • Update immediately
  • design and implement new negotiation protocols
  • design and implement new .sia format
  • achieve satisfactory test coverage
  • restore concurrency and expiration
    • restore multi-piece concurrency
    • restore multi-chunk concurrency
  • delete file data after expiration
    • (partial; protocol not finished)
  • implement data migration (for when too much redundancy is lost)
    • check for hosts going offline
    • check for hosts holding data hostage
    • check for hosts becoming too expensive
    • Note: contractor can't recover file data, only renter can
  • add persistence to hostdb
  • contractor needs to scan blockchain to confirm that contract txns and revision txns are submitted
    • needs to submit them itself if not submitted by host
    • can't make use of file contracts until they are seen in the blockchain

Unanswered questions:

  • Should contractor.Contracts() return all of the contracts, or just the most recent contract formed with each host? If not the latter, how do we avoid revising outdated contracts?
  • What do we do if we can't form contracts with enough hosts?

Merge request reports