Data-Availability Layer (DAL): DAL node store: index persistent data by slot id
Context
Previous work on garbage collection (%Data-Availability Layer (DAL): Garbage collection for shards storage) and legacy code cleanup (%Data-Availability Layer (DAL): DAL node: Legacy Code Clean-up & Workers Sketching) hinted at the need to index slots by slot id instead of commitment in the store. Indeed, garbage collection requires to refer to old slots by their publication level (see in particular #7124 (closed)) and replacing Irmin by the DAL node's KVS store requires to index data by continuous integers.
In %Data-Availability Layer (DAL): DAL node: Rehaul of the RPC interface, we adapted the RPC interface of the DAL node to allow this change of the storage, this milestone is about actually changing the layout of the storage to use slot ids instead of commitments in Irmin paths.
But before adapting the store, we need to introduce commitment-indexed memory caches for the data which have not yet been assigned a slot id and which need to be remembered until the publication of the commitment: in addition to the existing cache of shard proofs, we need a cache of slot contents and a cache of shards.
Work breakdown
-
Introduction of commitment-indexed caches
-
Add a cache for shards !13167 (merged) -
Use it to avoid writing shards on disk before commitment publication !13211 (merged) -
Add a cache for slots !12884 (merged) -
Use it to avoid writing slots on disk before commitment publication !13209 (merged)
-
-
Change of layout for the store
-
Legacy store: remove the mapping from slot ids to commitments !13217 (merged) -
Legacy store: remove the mapping from commitments to slot ids (path /commitments/<commitment>/headers/<slot_id>
) !13216 (merged) -
Store: index the slot store by slot ids !13212 (merged) -
Store: index the shard store by slot ids !13210 (merged)
-
-
Replace Irmin by the KVS !13224 (merged)
-
Add a KVS for slot status -
Use it instead of path /levels/<level>/<slot_index>/status
-
Remove all remaining traces of the legacy store
-
Work for a future milestone
- GC: this milestone should fix #7124 (closed), adding a test showing that this is fixed is however out of scope
Deliverables
-
A DAL node which does not depend on Irmin