Skip to content

Contractor journal

Luke Champine requested to merge log into master

My biggest PR in a while.

Recently it became clear that the contractor was not scaling well when storing lots of data. This was because it re-marshaled its entire persist data on every save -- and we save every time we revise a contract. Our tests showed that this amounted to about 35MB of data being encoded and written to disk (with fsync).

To address this, I wrote a json-based transaction journal. The journal records updates to the contracts instead of rewriting the whole contract. These updates are applied to the original object at regular intervals. The primary goal is to reduce latency when uploading/downloading, so for now, only those actions take advantage of the journal.

Things got a little ugly because the contractor persist object changed: it now uses string-keyed maps instead of slices in a few places.

Left to do:

  • fix bugs in journal (see TODOs in journal.go)
  • add metadata to journal format
  • test extensively on antfarm before even attempting with real contracts

Merge request reports