Create dotsia package
dotsia
is a package that defines the (new) .sia format and provides functions for encoding/decoding them.
The full specification is in format.go. Here's some design decisions I made:
- After favoring .zip for a while, I finally decide to switch to .tar.gz (my original choice). While .zip is convenient in some aspects, the tradeoffs weren't worth it. Specifically, .zip compresses each file individually, which is a huge loss since all of our files have the same structure. Also, reading a .zip requires that you know the size in advance (I know, right??).
- I chose not to use any Sia-specific types in the package, to make things as straightforward as possible. This decision may seem a little strange, but hear me out: doing so reduces coupling between the format and the Sia repo. For example, I could have used a
modules.NetAddress
instead of astring
for thehostAddress
field. But if we ever decided to add a marshalling method toNetAddress
(unlikely, I know), it could change how thedotsia
package encoded that field, which would invisibly break compatibility. Besides, fewer dependencies is always a nice thing. - I waffled on what to do about the
MasterKey
andErasureCode
fields. They are tricky because they need to be generic, which means you can't handle them inside thedotsia
package. Ideally you would first decode thename
field and branch on that, but JSON decodes everything at once. You could usejson.RawMessage
to get around this, but it's a pain for callers to work with. Ultimately I just stuck with the default type ofmap[string]interface{}
. Since these fields will generally be dead-simple anyway, it shouldn't be too painful to do the necessary typecasting. - The memory layout of the
File
struct has not changed much, despite the potential savings described in #1025 (closed). I think it's favorable to simply rely on compression to minimize any structural differences. It would be prudent, though, to compare different compressed structures with real-world data. - I put the new package inside the
modules/renter
folder, but it could alternatively live at the top level, or even in its own repo. This might make sense given its decoupling from the rest of the Sia code, but I'm not sure.
Next step is to integrate this package with the renter, replacing most of its persist code in the process. (Actually, next step is to get this branch building...)