perf(bifrost): round 2 — contract checks, stream buffers, HTX gzip, closeStream, Tx.Hash
Stacked on !5222 (merged) (targets its branch
bifrost-pprof-perf), so the diff below is only this round. Retarget todeveloponce !5222 (merged) merges.
Summary
This is a second, independent pass over the same 120 s production bifrost profile (cpu, allocs, heap, goroutine). This time the analysis came from a fresh agent with no knowledge of round 1's reasoning, and it was given the three already-open MRs so it would not repeat them. Every candidate was then re-checked against the code, under the same hard rule: no behavior changes at all. Five survived. The rest are listed under follow-ups, with the exact way each one would differ.
| Commit | Change | Profile evidence |
|---|---|---|
test(bifrost): benchmarks for round-2 … |
benchmarks first (_test.go only) |
— |
perf(bifrost): compare contract addresses as lowercase hex … |
EVM + ETH isToValidContractAddress stop computing the EIP-55 checksum (a keccak256) per call, or per list entry on ETH |
Address.String 0.41 s, 0.31 s of it from these two functions |
perf(bifrost): delay closeStream's Reset with a timer … |
time.AfterFunc instead of a goroutine sleeping 10 s |
2,321 of 3,177 goroutines (73 %) parked in that sleep |
perf(bifrost): reuse one gzip reader for HTX … |
one gzip.Reader, Reset per message |
gzip.NewReader 1.17 GB of lifetime allocations |
perf(bifrost): pool the p2p stream helpers' bufio … |
sync.Pool for the per-message 4 KiB bufio reader/writer |
bufio.NewReader + NewWriter 2.1 GB of lifetime allocations |
perf(common): build Tx.Hash without fmt |
string concatenation + an uppercase hex table instead of two Sprintfs |
Tx.Hash 0.24 s (0.22 s of it in fmt), from observedTxKey on every attestation |
Expected production effect: modest. Roughly 0.4 s of the 48.6 s of CPU samples (~0.8 %) is removed directly. About 3 MB/s less allocation (of ~55 MB/s) also means proportionally fewer GC cycles, and GC is ~10 % of CPU. On top of that: ~2,300 fewer goroutines and ~3 MB less memory held by pending stream resets. The larger remaining wins can't be shown exactly equivalent (see follow-ups).
Profile findings (fresh analysis, verified)
- CPU: 48.61 s over 120 s. 30 % is
encoding/json, almost all go-ethereum RPC decoding (left alone, as in round 1). 18.5 % is syscalls and cgo. 8 % is crypto (sender recovery, already done in !5222 (merged); gossipVerifySignature, kept). GC and allocation together are ~10–12 %. Bifrost's own code is only 1.2 % flat: the cost is in what it calls. - Allocations (
allocs.pprofcovers ~18 min of uptime: 59.9 GB, 481 M objects, ~55 MB/s):- library and startup: geth decoder refill and
RawMessage; tss-lib big.Int; argon2 at keystore unlock; - fixed in round 1:
fmt.Sprintf3.4 GB, 95 % of it fromGetAddress; - bifrost-side and fixed here:
bufio2.1 GB, HTX gzip 1.17 GB,Address.Hex13.7 M objects.
- library and startup: geth decoder refill and
- Heap: 290 MB in use, mostly leveldb buffer pools and memdbs across 15 databases. All are bounded by settings, and there is no sign of unbounded growth in this one snapshot.
- Goroutines: 3,177. 2,321 are
closeStream.func1sleeping 10 s; then ~113 per peer each for yamux and identify; 109 are ETHextractTxsworkers parked on its semaphore. No lock contention.
Equivalence arguments
1. Lowercase hex instead of the checksummed address
- The lowercase and EIP-55 forms differ only in the ASCII case of
a–f.strings.EqualFoldcompares rune by rune under case folding, soEqualFold(x, lower) == EqualFold(x, checksum)for anyx, including non-ASCII strings. LowerHexAddress(a)is exactlystrings.ToLower(a.Hex()). That is fuzzed, and checked on 10k random addresses.- The router and whitelist lists are now looped in turn instead of being
appended into a new slice. The result is the same any-match. - ETH:
addrwas dereferenced only when there was something to compare. The new code derives the string at the same point, so a niladdrpanics in exactly the same cases (tested). Production callers never pass nil. The panic text differs (a nil dereference instead of Go's "value method called using nil pointer"), on a path that can't be reached. - Differential tests: both functions are compared against verbatim copies of the originals over 20k random cases each:
- list entries in checksum, lower and upper case, with the
0xprefix missing, padded, truncated, and with Cyrillic/non-ASCII look-alikes; - whitelist on and off, plus ETH's global whitelist toggle;
- nil
addr.
- list entries in checksum, lower and upper case, with the
- Mutation check: dropping the whitelist loop, or computing the ETH string eagerly, fails these tests.
2. closeStream
time.AfterFunc(10s, f) runs f (the same Reset and the same error log) on its own goroutine at least 10 s after the call. That is the same guarantee as the old goroutine's time.Sleep(10s). Nothing ever waited on that goroutine. The only difference is that no goroutine sits parked for 10 s per closed stream.
3. HTX gzip reader
- In the Go source,
gzip.NewReader(r)isz := new(Reader); z.Reset(r). Resetassigns a freshReader{}, keeping only the decompressor.- flate's
Resetlikewise rebuilds its state, keeping buffer capacity. Its window is re-initialised withwrPos = 0, so back-references past what the current stream wrote are rejected, exactly as with a new window. No bytes can leak between messages. HandleWsMessageis called only from the provider's single websocket read loop, one message at a time.- Differential test: 424 messages, run twice (from a clean reader, and after leading bad input), go through one long-lived provider and, for each message, through a brand-new provider, whose first message takes exactly the old
NewReaderpath. Errors and tickers are compared. Inputs: valid, non-gzip, truncated, corrupt deflate, bad CRC, header name/comment/extra, the 10 KB read limit, multistream, empty input, and random bit flips and truncations. - Mutation check: skipping the per-message
Resetis caught at the second message.
4. p2p bufio pools
bufio.Reader.ResetandWriter.Resetrebuild exactly the state thatNewReader/NewWritergive a stream: the same 4 KiB buffer, and cleared read-ahead, unflushed bytes and sticky error. The stream therefore sees the sameRead/Writecalls with the same sizes, and callers get the same data and errors.NewReader's "already a*bufio.Reader" shortcut can't apply, because anetwork.Streamis never one.- Buffers are returned only after their last use. In the
WithContextvariants that is the inner goroutine, which can outlive a cancelled caller.Reset(nil)beforePutdrops the stream reference. - Differential test: 5,000 interleaved operations against verbatim copies (extracted from the base commit) of all four original helpers, comparing results and the bytes left on each stream. Pooled buffers carry state from failed uses into later ones. Cases:
- sizes around the 4 KiB boundary, up to 70 KB;
- chunked reads, back-to-back frames (read-ahead), truncated and oversized frames;
- writes failing mid-frame.
- Mutation check: handing out a pooled reader or writer without resetting it fails the test.
5. Tx.Hash (consensus path — please review with that in mind)
fmt.Sprintf("%s|%s|%s|%s|%d", …) calls each field's value-receiver String(). Chain, Address and Coins have no Format or Error methods, which fmt would prefer, and no String can panic: a zero-value Uint prints "<nil>" either way. %d of an int64 is strconv.FormatInt(…, 10), and %X of a [32]byte is the uppercase digits, high nibble first.
Thornode derives tx IDs from this on the consensus path (manager_pool_current.go:766, manager_network_current.go:1797), so:
- Differential test: 50k random txs against the original
fmtimplementation. Inputs: arbitrary and non-ASCII chains and addresses,|and%in fields, nil, empty and multi-coinCoins, zero-valueUints, synth/trade/secured assets, amounts up to 252 bits, and heights includingMinInt64/MaxInt64. - Mutation check: a lowercase hex table, unsigned height formatting, or hashing Gas instead of Coins each fails it.
- Wider tests:
go test -tags mocknet ./x/thorchain/passes.
Benchmarks
-count 10, -tags mocknet, AMD Ryzen Threadripper 3960X. The baseline is the benchmark commit, measured in a separate worktree.
BenchmarkIsToValidContractAddress: an AVAX-shaped list (2 routers + 17 aggregators).log/*is the per-log check the smart-contract log parser makes (includeWhiteList=false);tx/*is the per-tx check.ReceiptToTxInItem(from !5222 (merged)) is unchanged within noise, because its scanner has no routers registered and so never reaches these comparisons.- The ETH benchmark uses the mocknet whitelist (22 entries; mainnet has 43).
BenchmarkGCWithPendingStreamResetsholds the production count of 2,321 pending resets. GC time doesn't change measurably, because the test binary's own heap dominates it; the win is the memory the pending resets hold.
1. Lowercase-hex contract comparisons
│ base │ new │
│ sec/op │ sec/op vs base │
IsToValidContractAddress/log/miss-48 688.5n ± 3% 148.9n ± 6% -78.37% (p=0.000 n=10)
IsToValidContractAddress/log/hit-router-48 723.7n ± 2% 187.7n ± 4% -74.07% (p=0.000 n=10)
IsToValidContractAddress/tx/miss-48 1013.9n ± 8% 225.3n ± 2% -77.78% (p=0.000 n=10)
IsToValidContractAddress/tx/hit-last-aggregator-48 1297.5n ± 4% 462.9n ± 4% -64.33% (p=0.000 n=10)
ReceiptToTxInItem/NativeToVault/whitelist-48 76.78µ ± 1% 76.69µ ± 4% ~ (p=0.971 n=10)
ReceiptToTxInItem/NativeNonVault/whitelist-48 67.25µ ± 2% 66.53µ ± 4% ~ (p=0.075 n=10)
ReceiptToTxInItem/ContractCall/nowhitelist-48 74.67µ ± 3% 74.11µ ± 7% ~ (p=0.247 n=10)
ReceiptToTxInItem/DecodeOnly-48 1.926µ ± 2% 1.944µ ± 5% ~ (p=0.494 n=10)
geomean 5.139µ 2.608µ -49.25%
│ base │ new │
│ B/op │ B/op vs base │
IsToValidContractAddress/log/miss-48 160.00 ± 0% 80.00 ± 0% -50.00% (p=0.000 n=10)
IsToValidContractAddress/log/hit-router-48 160.00 ± 0% 80.00 ± 0% -50.00% (p=0.000 n=10)
IsToValidContractAddress/tx/miss-48 480.00 ± 0% 80.00 ± 0% -83.33% (p=0.000 n=10)
IsToValidContractAddress/tx/hit-last-aggregator-48 480.00 ± 0% 80.00 ± 0% -83.33% (p=0.000 n=10)
ReceiptToTxInItem/NativeToVault/whitelist-48 4.041Ki ± 0% 3.963Ki ± 0% -1.93% (p=0.000 n=10)
ReceiptToTxInItem/NativeNonVault/whitelist-48 1.845Ki ± 0% 1.767Ki ± 0% -4.24% (p=0.000 n=10)
ReceiptToTxInItem/ContractCall/nowhitelist-48 3.807Ki ± 0% 3.650Ki ± 0% -4.10% (p=0.000 n=10)
ReceiptToTxInItem/DecodeOnly-48 760.0 ± 0% 760.0 ± 0% ~ (p=1.000 n=10) ¹
geomean 779.7 413.5 -46.97%
│ base │ new │
│ allocs/op │ allocs/op vs base │
IsToValidContractAddress/log/miss-48 4.000 ± 0% 2.000 ± 0% -50.00% (p=0.000 n=10)
IsToValidContractAddress/log/hit-router-48 4.000 ± 0% 2.000 ± 0% -50.00% (p=0.000 n=10)
IsToValidContractAddress/tx/miss-48 5.000 ± 0% 2.000 ± 0% -60.00% (p=0.000 n=10)
IsToValidContractAddress/tx/hit-last-aggregator-48 5.000 ± 0% 2.000 ± 0% -60.00% (p=0.000 n=10)
ReceiptToTxInItem/NativeToVault/whitelist-48 110.0 ± 0% 108.0 ± 0% -1.82% (p=0.000 n=10)
ReceiptToTxInItem/NativeNonVault/whitelist-48 51.00 ± 0% 49.00 ± 0% -3.92% (p=0.000 n=10)
ReceiptToTxInItem/ContractCall/nowhitelist-48 91.00 ± 0% 87.00 ± 0% -4.40% (p=0.000 n=10)
ReceiptToTxInItem/DecodeOnly-48 19.00 ± 0% 19.00 ± 0% ~ (p=1.000 n=10) ¹
geomean 15.80 10.43 -33.98%
│ base │ new │
│ sec/op │ sec/op vs base │
ETHIsToValidContractAddress/log/miss-48 1282.5n ± 4% 124.4n ± 7% -90.30% (p=0.000 n=10)
ETHIsToValidContractAddress/log/hit-router-48 1343.0n ± 6% 162.2n ± 4% -87.92% (p=0.000 n=10)
ETHIsToValidContractAddress/tx/miss-48 15185.0n ± 2% 237.5n ± 2% -98.44% (p=0.000 n=10)
ETHIsToValidContractAddress/tx/hit-last-whitelisted-48 15343.0n ± 3% 282.1n ± 3% -98.16% (p=0.000 n=10)
geomean 4.476µ 191.8n -95.72%
│ base │ new │
│ B/op │ B/op vs base │
ETHIsToValidContractAddress/log/miss-48 288.00 ± 0% 80.00 ± 0% -72.22% (p=0.000 n=10)
ETHIsToValidContractAddress/log/hit-router-48 288.00 ± 0% 80.00 ± 0% -72.22% (p=0.000 n=10)
ETHIsToValidContractAddress/tx/miss-48 3488.00 ± 0% 80.00 ± 0% -97.71% (p=0.000 n=10)
ETHIsToValidContractAddress/tx/hit-last-whitelisted-48 3488.00 ± 0% 80.00 ± 0% -97.71% (p=0.000 n=10)
geomean 1002.3 80.00 -92.02%
│ base │ new │
│ allocs/op │ allocs/op vs base │
ETHIsToValidContractAddress/log/miss-48 7.000 ± 0% 2.000 ± 0% -71.43% (p=0.000 n=10)
ETHIsToValidContractAddress/log/hit-router-48 7.000 ± 0% 2.000 ± 0% -71.43% (p=0.000 n=10)
ETHIsToValidContractAddress/tx/miss-48 74.000 ± 0% 2.000 ± 0% -97.30% (p=0.000 n=10)
ETHIsToValidContractAddress/tx/hit-last-whitelisted-48 74.000 ± 0% 2.000 ± 0% -97.30% (p=0.000 n=10)
geomean 22.76 2.000 -91.21%2. closeStream timer
│ base │ new │
│ sec/op │ sec/op vs base │
GCWithPendingStreamResets-48 7.826m ± 1% 7.873m ± 2% ~ (p=0.481 n=10)
│ base │ new │
│ pending-B/stream │ pending-B/stream vs base │3. HTX gzip reader reuse
│ base │ new │
│ sec/op │ sec/op vs base │
HtxHandleWsMessage-48 27.14µ ± 10% 12.24µ ± 6% -54.90% (p=0.000 n=10)
│ base │ new │
│ B/op │ B/op vs base │
HtxHandleWsMessage-48 41.381Ki ± 0% 1.134Ki ± 0% -97.26% (p=0.000 n=10)
│ base │ new │
│ allocs/op │ allocs/op vs base │
HtxHandleWsMessage-48 26.00 ± 0% 21.00 ± 0% -19.23% (p=0.000 n=10)4. p2p bufio pools
│ base │ new │
│ sec/op │ sec/op vs base │
StreamRoundTrip-48 2849.0n ± 2% 798.2n ± 13% -71.98% (p=0.000 n=10)
│ base │ new │
│ B/op │ B/op vs base │
StreamRoundTrip-48 9.602Ki ± 0% 1.509Ki ± 0% -84.29% (p=0.000 n=10)
│ base │ new │
│ allocs/op │ allocs/op vs base │
StreamRoundTrip-48 6.000 ± 0% 3.000 ± 0% -50.00% (p=0.000 n=10)5. Tx.Hash without fmt
│ base │ new │
│ sec/op │ sec/op vs base │
TxHash-48 2.711µ ± 6% 1.081µ ± 24% -60.12% (p=0.000 n=10)
│ base │ new │
│ B/op │ B/op vs base │
TxHash-48 536.0 ± 0% 400.0 ± 0% -25.37% (p=0.000 n=10)
│ base │ new │
│ allocs/op │ allocs/op vs base │
TxHash-48 16.00 ± 0% 10.00 ± 0% -37.50% (p=0.000 n=10)Testing
go test -tags mockneton all touched packages passes, as on the base commit:common(also without the tag),bifrost/pkg/chainclients/{evm,ethereum,shared/evm},bifrost/observer,bifrost/oracle/...,bifrost/p2p/.... Alsox/thorchain.-race:bifrost/p2p, and the HTX test.FuzzLowerHexAddressran for 20 s.- golangci-lint v1.63.4 with the repo config on the touched packages reports only the pre-existing SA4006 in
common/address_mocknet_test.go. gofumpt is clean.go build ./...passes.
Follow-ups (not done: can't be shown exactly equivalent)
Items 1–6 are now in !5231 (merged), one commit each, with the behavior differences pinned by tests. Ranked by CPU. Each states exactly what would change.
RemoveSignedTxItem: checkHasbeforeDelete(~0.6–0.7 s, 1.3–1.4 %).- Today: every tx in every EVM block writes a leveldb delete marker for a signed-tx hash that was almost never stored (
getTxInOptimizedL385: 0.25 s). The unstuck loop'sGetSignedTxItemsthen iterates through those markers (0.38 s). - What would change: if the key is missing and leveldb is in a write-failure or read-only state, the call returns nil instead of the write error, and that log line goes away.
- Today: every tx in every EVM block writes a leveldb delete marker for a signed-tx hash that was almost never stored (
Keys.GetSignerInfo: cache the keyring record (~0.26 s, plus ~22 M objects, about 4.5 % of all allocations).- Today: every
FetchNodeStatus(about once a second viasendDeck) re-decrypts the keyring entry (jose2go + PBKDF2). - What would change: a keyring file edited at runtime wouldn't be re-read, and a transient read error would no longer panic.
- Today: every
- Tron
GetLatestBlock: decode only the block number (~0.2–0.3 s).- What would change: a wrongly typed non-number field would no longer fail the height lookup. That is the same trade as !5225 (merged), and
FetchTxsstill fully decodes the block.
- What would change: a wrongly typed non-number field would no longer fail the height lookup. That is the same trade as !5225 (merged), and
PruneBlockMeta: decode onlyheight(~0.15–0.25 s).- What would change: a stored entry with a mistyped non-height field would no longer abort the prune. Bifrost itself only ever writes well-typed entries.
- UTXO: reuse the block already scanned for the coinbase value. This halves the multi-MB
getblockRPCs on BTC, LTC, BCH, DOGE and ZEC.- What would change: it adds a cache, and it succeeds where a second fetch would have failed transiently.
getTxInOptimized: a worker pool instead of a goroutine per receipt (~0.2 s of stack growth).- What would change: a restructure of the concurrency, and it overlaps !5224 (merged).
Other ideas worth a decision:
eth_getBlockReceiptsfor ETH, which today makes one receipt RPC per tx, serialized at concurrency 1.- Turning off RPC gzip for nodes on a local network (gzip is 2.7 % of CPU).
GOGC/GOMEMLIMITtuning: the live heap is mostly pointer-free leveldb buffers.- Long-lived p2p streams instead of a stream per batch per peer (yamux costs 3.8 s; ~4,400 goroutines are created per second).
- Mimir
EVMDisableContractWhitelistread once per block instead of once per receipt. - MEXC: request only the needed symbols.
Not optimizations, found along the way (verified; the three bugs are fixed in !5224 (merged))
bifrost/pkg/chainclients/tron/api/tron_api.go:333:api.logger.Err(err)has no.Msg(), so this error is never logged. The error is still returned.bifrost/pkg/chainclients/evm/evm_block_scanner.go:534:errors.Is(err, ethereum.NotFound)tests the outererr, which is always nil there, instead ofelem.Error. On the batch-receipt path, NotFound receipts are always logged as errors.getTxInOptimizedblocks forever ifConcurrencyis 0 (unbuffered semaphore). Nothing validates it; every EVM config sets at least 1.- ETH
extractTxsstarts a goroutine for every tx before taking the semaphore, so ~200 sit parked per block at concurrency 1.