Skip to content

Explorer Database

Luke Champine requested to merge explorer-db into master

I know this was gonna be Jordan's job, but I got tired of restarting the explorer and banged it out in two days. Sorry.

Previously, siad -M cge consumed about 2GB of memory, causing it to crash on many systems. Notably, it was crashing http://explore.sia.tech, filling up both its 2GB physical RAM and 2GB swap.

As of this PR, the explorer database now resides on disk, in a bolt database. siad -M cge consumes less than 10MB of RAM (!!). I found out that htop may be misleading for judging memory usage; it seems to include the mmap overhead of bolt. (The explorer database is about 2GB.) pmap -x $(pidof siad) gives more accurate results.

I tried to preserve the old logic as much as possible, but it's quite likely that a discrepancy or two has snuck in. I know for sure that the difficulty calculation has changed, and likely the estimated hashrate as well. I would appreciate some assistance in calculating those values correctly. The way we handle FileContractDiffs is pretty gross too. My gut says there's a much better way to do it. The only major changes were to block facts. The old explorer kept a set of "current" facts, but the new model just maps each block to its facts. This broke one test (see TODO). Also, blockFacts now embeds a modules.BlockFacts. They only difference between the two is that blockFacts has an extra Timestamp field, and modules.BlockFacts has a redundant Difficulty field (can be calculated from the target).

Finally, I need to add code to handle subscription errors. Restarting from 0 seems sensible, but the existing database needs to be wiped first.

Oh yeah -- no new tests added. :I I've run it locally and confirmed that it functions as expected, but obviously some rigorous testing would be better.

ALSO: I think I may have identified what was "pushing the explorer over the edge" and causing it to crash. If you ask it about the zero hash, it will report every transaction containing the zero hash, which is... a lot. Even with the DB, memory and CPU usage spike for multiple minutes. We could triage this by intercepting the zero hash in the API, but I'm not sure what the best behavior is long-term.

Merge request reports