migration script: old barrel data to new barrel
With the arrival of the stable release we should prepare the migration of the data from an old barrel to the new barrel since some nodes are used around. To do it we will reuse the changes feed to dump the current data to the disk.
requirements
- document, revisions and their metadata (revision tree) should be stored as is
- sequences of documents should be preserved
- document ids should be preserved.
implementation
we stores docs on disc in an append only log. Documents are in the following form. Revisions are appended after the metadata.
----
METADATA DOC1 (DOCID, REVTREE, SEQ, DELETED, WINNING ID)
REV1
...
REV2
---
METATDATA DOC2
...
file format
At the beginning of the doc a MAGIC flag is added, followed by the version (32 bits integer):
1 32
<-----><------->
<MAGIC><VERSION>
Magic cookie is 16#01 .
We only then store the data as Erlang term in the following manner
1 32 32
<-------><----><----><---->.........
<DOCFLAG><size><META><size><revision>
<DOCFLAG><size><META><size><revision2>
....
DocFlag is 16#02
note: a possible implementation would be to split the log if it comes too large, but we can see it later.
general details
A module exposition the simple migrate
function will be added to barrel. Then the node script will be extended.
testing
This is the minimum testing suite we need to have on the migration script:
- test if the version is correctly appended.
- iterarte the file and retrieve docs