Skip to content

Create dotsia package

Luke Champine requested to merge dotsia into renter-host-upgrades

dotsia is a package that defines the (new) .sia format and provides functions for encoding/decoding them.

The full specification is in format.go. Here's some design decisions I made:

  • After favoring .zip for a while, I finally decide to switch to .tar.gz (my original choice). While .zip is convenient in some aspects, the tradeoffs weren't worth it. Specifically, .zip compresses each file individually, which is a huge loss since all of our files have the same structure. Also, reading a .zip requires that you know the size in advance (I know, right??).
  • I chose not to use any Sia-specific types in the package, to make things as straightforward as possible. This decision may seem a little strange, but hear me out: doing so reduces coupling between the format and the Sia repo. For example, I could have used a modules.NetAddress instead of a string for the hostAddress field. But if we ever decided to add a marshalling method to NetAddress (unlikely, I know), it could change how the dotsia package encoded that field, which would invisibly break compatibility. Besides, fewer dependencies is always a nice thing.
  • I waffled on what to do about the MasterKey and ErasureCode fields. They are tricky because they need to be generic, which means you can't handle them inside the dotsia package. Ideally you would first decode the name field and branch on that, but JSON decodes everything at once. You could use json.RawMessage to get around this, but it's a pain for callers to work with. Ultimately I just stuck with the default type of map[string]interface{}. Since these fields will generally be dead-simple anyway, it shouldn't be too painful to do the necessary typecasting.
  • The memory layout of the File struct has not changed much, despite the potential savings described in #1025 (closed). I think it's favorable to simply rely on compression to minimize any structural differences. It would be prudent, though, to compare different compressed structures with real-world data.
  • I put the new package inside the modules/renter folder, but it could alternatively live at the top level, or even in its own repo. This might make sense given its decoupling from the rest of the Sia code, but I'm not sure.

Next step is to integrate this package with the renter, replacing most of its persist code in the process. (Actually, next step is to get this branch building...)

Merge request reports