Explorer Database
I know this was gonna be Jordan's job, but I got tired of restarting the explorer and banged it out in two days. Sorry.
Previously, siad -M cge
consumed about 2GB of memory, causing it to crash on many systems. Notably, it was crashing http://explore.sia.tech, filling up both its 2GB physical RAM and 2GB swap.
As of this PR, the explorer database now resides on disk, in a bolt database. siad -M cge
consumes less than 10MB of RAM (!!). I found out that htop
may be misleading for judging memory usage; it seems to include the mmap overhead of bolt. (The explorer database is about 2GB.) pmap -x $(pidof siad)
gives more accurate results.
I tried to preserve the old logic as much as possible, but it's quite likely that a discrepancy or two has snuck in. I know for sure that the difficulty calculation has changed, and likely the estimated hashrate as well. I would appreciate some assistance in calculating those values correctly. The way we handle FileContractDiffs is pretty gross too. My gut says there's a much better way to do it.
The only major changes were to block facts. The old explorer kept a set of "current" facts, but the new model just maps each block to its facts. This broke one test (see TODO). Also, blockFacts
now embeds a modules.BlockFacts
. They only difference between the two is that blockFacts
has an extra Timestamp
field, and modules.BlockFacts
has a redundant Difficulty
field (can be calculated from the target).
Finally, I need to add code to handle subscription errors. Restarting from 0 seems sensible, but the existing database needs to be wiped first.
Oh yeah -- no new tests added. :I I've run it locally and confirmed that it functions as expected, but obviously some rigorous testing would be better.
ALSO: I think I may have identified what was "pushing the explorer over the edge" and causing it to crash. If you ask it about the zero hash, it will report every transaction containing the zero hash, which is... a lot. Even with the DB, memory and CPU usage spike for multiple minutes. We could triage this by intercepting the zero hash in the API, but I'm not sure what the best behavior is long-term.