Reduce Memory usage of TxIndex
The goal of this PR is to reduce the memory used by the TxIndex.
A little bit of context
The key problem is that we are keeping a lot of transactions in the index indefinitely. The TxIndex used to expose an unregister-function. But this relies on processes cleaning up after themselves which isn't happening (yet).
To make matters worse the full TxIndex is stored in memory. Even on signet which has little usage we observe high memory usage which can crash the Ark Server.
An obvious approach is to just ensure we call unregister whenever we don't need the data anymore. This approach has a few problems.
- It is very hard to debug. Was it the forfeit watcher, sweeper, or broadcaster that forgot to clean-up properly.
- Multiple processes interact. Eg: The broadcaster puts a round transaction in the index and the sweeper expects to find every successful round
I decided to go for a little bit of a different approach. The TxIndex has been renamed to TxIndexData to clarify that it is just an in-memory data-structure.
All processes are now relying on the TxCache. The cache will first try and find transactions in the local TxIndexData. If the transaction cannot be found it will get the tx from the database and the corresponding status from bitcoind.
What I tried to preserve?
I tried to keep as much of the API intact.
- Most functions stays the same
- You can still query the TxStatus infallible
However, adding a transaction to the TxCache is not invalid anymore. It will only succeed if we manage to also write that transaction to the database.