Skip to content

[tune] Bifrost LevelDB Storage

Ursa (9R) requested to merge ursa/leveldb-tuning into develop

We've observed a significant increase in Bifrost CPU usage since the growth of the Bitocin mempool. Upon profiling an active Bifrost it was determined that the majority of this load was due to existence checks on mempool transactions in LevelDB. LSM trees (LevelDB) are a particularly bad data structure for inclusion checks with a high positive rate - the bloom filters offset read amplification on non-existent keys (good for low positive rate), but existent keys still require table scan to confirm. Once you blow throught the user space table cache it requires disk reads and decompression for all sstables with a positive result from the filter.

image__1_

This PR exposes parameters for tuning LevelDB storage options as well as the addition of an optional LRU cache for transaction IDs in the scanner storage. On stagenet we've observed a 2-3x reduction (green line below) in average load with these settings (mempool observation latency should improve as well, but to a lesser degree).

image__2_

The only changes that will roll out to all nodes are bloom filters on all LevelDB instances. We'll monitor the following tuning options on Nine Realms mainnet validators before proposing as the defaults in the next release:

BIFROST_CHAINS_BTC_MEMPOOL_TX_ID_CACHE_SIZE: "1_000_000"                  # current ~150k
BIFROST_CHAINS_BTC_SCANNER_LEVELDB_BLOCK_CACHE_CAPACITY: "536870912"      # 512 Mb
BIFROST_CHAINS_BTC_SCANNER_LEVELDB_COMPACTION_TABLE_SIZE_MULTIPLIER: "10" # 20 Mb tables
BIFROST_CHAINS_BTC_SCANNER_LEVELDB_WRITE_BUFFER: "20971520"               # match table

# compact on init for all leveldb storage
BIFROST_OBSERVER_LEVELDB_COMPACT_ON_INIT: "true"
BIFROST_SIGNER_LEVELDB_COMPACT_ON_INIT: "true"
BIFROST_CHAINS_AVAX_SCANNER_LEVELDB_COMPACT_ON_INIT: "true"
BIFROST_CHAINS_BCH_SCANNER_LEVELDB_COMPACT_ON_INIT: "true"
BIFROST_CHAINS_BNB_SCANNER_LEVELDB_COMPACT_ON_INIT: "true"
BIFROST_CHAINS_BTC_SCANNER_LEVELDB_COMPACT_ON_INIT: "true"
BIFROST_CHAINS_DOGE_SCANNER_LEVELDB_COMPACT_ON_INIT: "true"
BIFROST_CHAINS_ETH_SCANNER_LEVELDB_COMPACT_ON_INIT: "true"
BIFROST_CHAINS_GAIA_SCANNER_LEVELDB_COMPACT_ON_INIT: "true"
BIFROST_CHAINS_LTC_SCANNER_LEVELDB_COMPACT_ON_INIT: "true"

Merge request reports