Skip to content

Change semantics of index-batch-size

Dagur Valberg Johannsson requested to merge github/fork/dagurval/block-batches into master

Created by: dagurval

The old semantics is to request 'index-batch-size' number of blocks in one RPC request.

This sometimes causes a very large peak memory usage in bitcoind, as index-batch-size number of potentially 32MB blocks need to be serialized as JSON in one request. See issue #3 (closed).

The new semantics is add a sync buffer of size 'index-batch-size' to the electrum server and continuously try to fill the buffer by fetching one block at a time.

Test plan

Full reindex of mainnet. Observe performance is not noticably impacted.

At around block height 360k, performance is at > 20 blocks/s.

$ tail -f debug.log | grep --line-buffered getblock | pv --line-mode -b --average-rate --rate > /dev/null
0.8k [18,7 /s] [20,7 /s]

(Requires with `debug=rpc` in `bitcoin.conf`)

Merge request reports