File attributes

MERGE REQUEST

Overview

Add Attributes to SiaFile

Changes to Work Flow

Amazon S3 allows object metadata (see here) to be stored in key-value pairs. Their implementation is divided into 2 categories: System-Defined Object Metadata and User-Defined Object Metadata. This MR adds the latter to Sia (User-Defined Object Metadata). Instead of straight key-value pair storage, I extended this by implementing a namespace for per-application separation - a best effort attempt at preventing key collisions.

Since Sia's equivalent of System-Defined Object Metadata is currently stored in '.sia' files as json data, I added the following additional structure:

{
  "attributes": {
    "<namespace>": {
	    "<key>": "<value>",
	    "<key>": "<value>",
        ...
	}
 }
}

S3 imposes some constraints on sizes, specifically key length and total size of user-defined key-value data. Key names can be no longer than 1024 bytes and total size of data must not exceed 2KB. I've extended this to allow multiple namespaces to consume a total of 16KB, but each namespace is limited to 2KB.

Additionally, there are restrictions on characters that can/cannot be used in key names (extended to namespace names). These can be found here under Object Key Naming Guidelines. To remain along the lines of S3-compatibility, I kept this the same as well.

I also added the following new endpoints for attribute manipulation:

• /renter/attributes/siapath [GET]
• /renter/attributes/siapath [POST]

CLI

siac has also been modified to include attribute retrieval/manipulation. The following new CLI options have been added:

• siac renter attributes [siapath] [namespace] [key]
• siac renter setattributes [siapath] [namespace] <[key] [value]> <[attributes map]>

Questions / TODOs

• Should these attributes be returned by default when calling /renter/dir, /renter/file, and /renter/files or should this data only be returned using the newly added endpoints?
  • I personally suggest that attribute data return be restricted to the new enpoints to keep the existing dataset as small as possible, but also see the benefit of just having this data readily available.

• Directory attributes will not be supported but are probably advisable. May need to be implemented at a later time.

Testing Changes

modules/renter/files_test.go

• Added TestRenterAttributes()

siatest/renter/renter_test.go

• Added {"TestAttributes", testAttributes} to TestRenterTwo()

Documentation Changes

• Updated doc/api/index.html.md to include:
  • /renter/attributes/siapath [GET]
  • /renter/attributes/siapath [POST]

Modules and Packages Impacted

• modules/renter/siafile
• modules/renter

Example for Visual changes (ie Screenshot)

• Add visual proof of siac commands working. asciinema is great for this
  • siac.cast
• Add screen shot of the api doc markdown to confirm the formatting looks good.
  • 

Issues Closed

Checklist

Review and complete the checklist to ensure that the MR is complete before assigned to an approver.

• All new methods, or updating methods have clear docstrings
• Testing added or updated for new methods
• API documentation updated for API updates
• Module README.md updated for changes to work flow
• Issue added to Sia-UI repo for new supporting features