S3 Backend
The S3 Backend should use the same file format as the multifile backend, but should use name prefixes instead of directories.
MultiFile Backend File Name | S3 File Name |
---|---|
key |
key |
data/X/0 |
data-0 |
data/X/0.header |
data-0.header |
index/0 |
index-0 |
mainfest/0 |
manifest-0 |
mainfest/chunk.settings |
manifest-chunk.settings |
readlocks-(uuid) |
readlock-(uuid) |
There should also probably be a program (or a command in asuran-cli) to convert a raw copy of an S3-backend repository into a valid MultiFile-backend repository, and vice versa, to convert a MultiFile-Backend repository into one suitable for direct upload to S3. This should be a fairly safe operation as it is accomplished entirely through file renames.
This will be a bit more involved, as the current abstractions for building MultiFile-like backends require Read/Seek/Write implementations, which doesn't map well to S3. Ideally we should get some reusable abstractions out of this.
Edited by Nathan McCarty