perf(bifrost): round 2 — contract checks, stream buffers, HTX gzip, closeStream, Tx.Hash

Stacked on !5222 (merged) (targets its branch bifrost-pprof-perf), so the diff below is only this round. Retarget to develop once !5222 (merged) merges.

Summary

This is a second, independent pass over the same 120 s production bifrost profile (cpu, allocs, heap, goroutine). This time the analysis came from a fresh agent with no knowledge of round 1's reasoning, and it was given the three already-open MRs so it would not repeat them. Every candidate was then re-checked against the code, under the same hard rule: no behavior changes at all. Five survived. The rest are listed under follow-ups, with the exact way each one would differ.

Commit Change Profile evidence
test(bifrost): benchmarks for round-2 … benchmarks first (_test.go only) —
perf(bifrost): compare contract addresses as lowercase hex … EVM + ETH isToValidContractAddress stop computing the EIP-55 checksum (a keccak256) per call, or per list entry on ETH Address.String 0.41 s, 0.31 s of it from these two functions
perf(bifrost): delay closeStream's Reset with a timer … time.AfterFunc instead of a goroutine sleeping 10 s 2,321 of 3,177 goroutines (73 %) parked in that sleep
perf(bifrost): reuse one gzip reader for HTX … one gzip.Reader, Reset per message gzip.NewReader 1.17 GB of lifetime allocations
perf(bifrost): pool the p2p stream helpers' bufio … sync.Pool for the per-message 4 KiB bufio reader/writer bufio.NewReader + NewWriter 2.1 GB of lifetime allocations
perf(common): build Tx.Hash without fmt string concatenation + an uppercase hex table instead of two Sprintfs Tx.Hash 0.24 s (0.22 s of it in fmt), from observedTxKey on every attestation

Expected production effect: modest. Roughly 0.4 s of the 48.6 s of CPU samples (~0.8 %) is removed directly. About 3 MB/s less allocation (of ~55 MB/s) also means proportionally fewer GC cycles, and GC is ~10 % of CPU. On top of that: ~2,300 fewer goroutines and ~3 MB less memory held by pending stream resets. The larger remaining wins can't be shown exactly equivalent (see follow-ups).

Profile findings (fresh analysis, verified)

  • CPU: 48.61 s over 120 s. 30 % is encoding/json, almost all go-ethereum RPC decoding (left alone, as in round 1). 18.5 % is syscalls and cgo. 8 % is crypto (sender recovery, already done in !5222 (merged); gossip VerifySignature, kept). GC and allocation together are ~10–12 %. Bifrost's own code is only 1.2 % flat: the cost is in what it calls.
  • Allocations (allocs.pprof covers ~18 min of uptime: 59.9 GB, 481 M objects, ~55 MB/s):
    • library and startup: geth decoder refill and RawMessage; tss-lib big.Int; argon2 at keystore unlock;
    • fixed in round 1: fmt.Sprintf 3.4 GB, 95 % of it from GetAddress;
    • bifrost-side and fixed here: bufio 2.1 GB, HTX gzip 1.17 GB, Address.Hex 13.7 M objects.
  • Heap: 290 MB in use, mostly leveldb buffer pools and memdbs across 15 databases. All are bounded by settings, and there is no sign of unbounded growth in this one snapshot.
  • Goroutines: 3,177. 2,321 are closeStream.func1 sleeping 10 s; then ~113 per peer each for yamux and identify; 109 are ETH extractTxs workers parked on its semaphore. No lock contention.

Equivalence arguments

1. Lowercase hex instead of the checksummed address

  • The lowercase and EIP-55 forms differ only in the ASCII case of a–f. strings.EqualFold compares rune by rune under case folding, so EqualFold(x, lower) == EqualFold(x, checksum) for any x, including non-ASCII strings.
  • LowerHexAddress(a) is exactly strings.ToLower(a.Hex()). That is fuzzed, and checked on 10k random addresses.
  • The router and whitelist lists are now looped in turn instead of being appended into a new slice. The result is the same any-match.
  • ETH: addr was dereferenced only when there was something to compare. The new code derives the string at the same point, so a nil addr panics in exactly the same cases (tested). Production callers never pass nil. The panic text differs (a nil dereference instead of Go's "value method called using nil pointer"), on a path that can't be reached.
  • Differential tests: both functions are compared against verbatim copies of the originals over 20k random cases each:
    • list entries in checksum, lower and upper case, with the 0x prefix missing, padded, truncated, and with Cyrillic/non-ASCII look-alikes;
    • whitelist on and off, plus ETH's global whitelist toggle;
    • nil addr.
  • Mutation check: dropping the whitelist loop, or computing the ETH string eagerly, fails these tests.

2. closeStream

time.AfterFunc(10s, f) runs f (the same Reset and the same error log) on its own goroutine at least 10 s after the call. That is the same guarantee as the old goroutine's time.Sleep(10s). Nothing ever waited on that goroutine. The only difference is that no goroutine sits parked for 10 s per closed stream.

3. HTX gzip reader

  • In the Go source, gzip.NewReader(r) is z := new(Reader); z.Reset(r).
  • Reset assigns a fresh Reader{}, keeping only the decompressor.
  • flate's Reset likewise rebuilds its state, keeping buffer capacity. Its window is re-initialised with wrPos = 0, so back-references past what the current stream wrote are rejected, exactly as with a new window. No bytes can leak between messages.
  • HandleWsMessage is called only from the provider's single websocket read loop, one message at a time.
  • Differential test: 424 messages, run twice (from a clean reader, and after leading bad input), go through one long-lived provider and, for each message, through a brand-new provider, whose first message takes exactly the old NewReader path. Errors and tickers are compared. Inputs: valid, non-gzip, truncated, corrupt deflate, bad CRC, header name/comment/extra, the 10 KB read limit, multistream, empty input, and random bit flips and truncations.
  • Mutation check: skipping the per-message Reset is caught at the second message.

4. p2p bufio pools

  • bufio.Reader.Reset and Writer.Reset rebuild exactly the state that NewReader/NewWriter give a stream: the same 4 KiB buffer, and cleared read-ahead, unflushed bytes and sticky error. The stream therefore sees the same Read/Write calls with the same sizes, and callers get the same data and errors.
  • NewReader's "already a *bufio.Reader" shortcut can't apply, because a network.Stream is never one.
  • Buffers are returned only after their last use. In the WithContext variants that is the inner goroutine, which can outlive a cancelled caller. Reset(nil) before Put drops the stream reference.
  • Differential test: 5,000 interleaved operations against verbatim copies (extracted from the base commit) of all four original helpers, comparing results and the bytes left on each stream. Pooled buffers carry state from failed uses into later ones. Cases:
    • sizes around the 4 KiB boundary, up to 70 KB;
    • chunked reads, back-to-back frames (read-ahead), truncated and oversized frames;
    • writes failing mid-frame.
  • Mutation check: handing out a pooled reader or writer without resetting it fails the test.

5. Tx.Hash (consensus path — please review with that in mind)

fmt.Sprintf("%s|%s|%s|%s|%d", …) calls each field's value-receiver String(). Chain, Address and Coins have no Format or Error methods, which fmt would prefer, and no String can panic: a zero-value Uint prints "<nil>" either way. %d of an int64 is strconv.FormatInt(…, 10), and %X of a [32]byte is the uppercase digits, high nibble first.

Thornode derives tx IDs from this on the consensus path (manager_pool_current.go:766, manager_network_current.go:1797), so:

  • Differential test: 50k random txs against the original fmt implementation. Inputs: arbitrary and non-ASCII chains and addresses, | and % in fields, nil, empty and multi-coin Coins, zero-value Uints, synth/trade/secured assets, amounts up to 252 bits, and heights including MinInt64/MaxInt64.
  • Mutation check: a lowercase hex table, unsigned height formatting, or hashing Gas instead of Coins each fails it.
  • Wider tests: go test -tags mocknet ./x/thorchain/ passes.

Benchmarks

-count 10, -tags mocknet, AMD Ryzen Threadripper 3960X. The baseline is the benchmark commit, measured in a separate worktree.

  • BenchmarkIsToValidContractAddress: an AVAX-shaped list (2 routers + 17 aggregators). log/* is the per-log check the smart-contract log parser makes (includeWhiteList=false); tx/* is the per-tx check.
  • ReceiptToTxInItem (from !5222 (merged)) is unchanged within noise, because its scanner has no routers registered and so never reaches these comparisons.
  • The ETH benchmark uses the mocknet whitelist (22 entries; mainnet has 43).
  • BenchmarkGCWithPendingStreamResets holds the production count of 2,321 pending resets. GC time doesn't change measurably, because the test binary's own heap dominates it; the win is the memory the pending resets hold.

1. Lowercase-hex contract comparisons

                                                   │     base     │                 new                 │
                                                   │    sec/op    │   sec/op     vs base                │
IsToValidContractAddress/log/miss-48                  688.5n ± 3%   148.9n ± 6%  -78.37% (p=0.000 n=10)
IsToValidContractAddress/log/hit-router-48            723.7n ± 2%   187.7n ± 4%  -74.07% (p=0.000 n=10)
IsToValidContractAddress/tx/miss-48                  1013.9n ± 8%   225.3n ± 2%  -77.78% (p=0.000 n=10)
IsToValidContractAddress/tx/hit-last-aggregator-48   1297.5n ± 4%   462.9n ± 4%  -64.33% (p=0.000 n=10)
ReceiptToTxInItem/NativeToVault/whitelist-48          76.78µ ± 1%   76.69µ ± 4%        ~ (p=0.971 n=10)
ReceiptToTxInItem/NativeNonVault/whitelist-48         67.25µ ± 2%   66.53µ ± 4%        ~ (p=0.075 n=10)
ReceiptToTxInItem/ContractCall/nowhitelist-48         74.67µ ± 3%   74.11µ ± 7%        ~ (p=0.247 n=10)
ReceiptToTxInItem/DecodeOnly-48                       1.926µ ± 2%   1.944µ ± 5%        ~ (p=0.494 n=10)
geomean                                               5.139µ        2.608µ       -49.25%

                                                   │     base     │                  new                   │
                                                   │     B/op     │     B/op      vs base                  │
IsToValidContractAddress/log/miss-48                  160.00 ± 0%     80.00 ± 0%  -50.00% (p=0.000 n=10)
IsToValidContractAddress/log/hit-router-48            160.00 ± 0%     80.00 ± 0%  -50.00% (p=0.000 n=10)
IsToValidContractAddress/tx/miss-48                   480.00 ± 0%     80.00 ± 0%  -83.33% (p=0.000 n=10)
IsToValidContractAddress/tx/hit-last-aggregator-48    480.00 ± 0%     80.00 ± 0%  -83.33% (p=0.000 n=10)
ReceiptToTxInItem/NativeToVault/whitelist-48         4.041Ki ± 0%   3.963Ki ± 0%   -1.93% (p=0.000 n=10)
ReceiptToTxInItem/NativeNonVault/whitelist-48        1.845Ki ± 0%   1.767Ki ± 0%   -4.24% (p=0.000 n=10)
ReceiptToTxInItem/ContractCall/nowhitelist-48        3.807Ki ± 0%   3.650Ki ± 0%   -4.10% (p=0.000 n=10)
ReceiptToTxInItem/DecodeOnly-48                        760.0 ± 0%     760.0 ± 0%        ~ (p=1.000 n=10) ¹
geomean                                                779.7          413.5       -46.97%

                                                   │    base    │                 new                  │
                                                   │ allocs/op  │ allocs/op   vs base                  │
IsToValidContractAddress/log/miss-48                 4.000 ± 0%   2.000 ± 0%  -50.00% (p=0.000 n=10)
IsToValidContractAddress/log/hit-router-48           4.000 ± 0%   2.000 ± 0%  -50.00% (p=0.000 n=10)
IsToValidContractAddress/tx/miss-48                  5.000 ± 0%   2.000 ± 0%  -60.00% (p=0.000 n=10)
IsToValidContractAddress/tx/hit-last-aggregator-48   5.000 ± 0%   2.000 ± 0%  -60.00% (p=0.000 n=10)
ReceiptToTxInItem/NativeToVault/whitelist-48         110.0 ± 0%   108.0 ± 0%   -1.82% (p=0.000 n=10)
ReceiptToTxInItem/NativeNonVault/whitelist-48        51.00 ± 0%   49.00 ± 0%   -3.92% (p=0.000 n=10)
ReceiptToTxInItem/ContractCall/nowhitelist-48        91.00 ± 0%   87.00 ± 0%   -4.40% (p=0.000 n=10)
ReceiptToTxInItem/DecodeOnly-48                      19.00 ± 0%   19.00 ± 0%        ~ (p=1.000 n=10) ¹
geomean                                              15.80        10.43       -33.98%

                                                       │     base      │                 new                 │
                                                       │    sec/op     │   sec/op     vs base                │
ETHIsToValidContractAddress/log/miss-48                   1282.5n ± 4%   124.4n ± 7%  -90.30% (p=0.000 n=10)
ETHIsToValidContractAddress/log/hit-router-48             1343.0n ± 6%   162.2n ± 4%  -87.92% (p=0.000 n=10)
ETHIsToValidContractAddress/tx/miss-48                   15185.0n ± 2%   237.5n ± 2%  -98.44% (p=0.000 n=10)
ETHIsToValidContractAddress/tx/hit-last-whitelisted-48   15343.0n ± 3%   282.1n ± 3%  -98.16% (p=0.000 n=10)
geomean                                                    4.476µ        191.8n       -95.72%

                                                       │     base     │                new                 │
                                                       │     B/op     │    B/op     vs base                │
ETHIsToValidContractAddress/log/miss-48                   288.00 ± 0%   80.00 ± 0%  -72.22% (p=0.000 n=10)
ETHIsToValidContractAddress/log/hit-router-48             288.00 ± 0%   80.00 ± 0%  -72.22% (p=0.000 n=10)
ETHIsToValidContractAddress/tx/miss-48                   3488.00 ± 0%   80.00 ± 0%  -97.71% (p=0.000 n=10)
ETHIsToValidContractAddress/tx/hit-last-whitelisted-48   3488.00 ± 0%   80.00 ± 0%  -97.71% (p=0.000 n=10)
geomean                                                   1002.3        80.00       -92.02%

                                                       │    base     │                new                 │
                                                       │  allocs/op  │ allocs/op   vs base                │
ETHIsToValidContractAddress/log/miss-48                   7.000 ± 0%   2.000 ± 0%  -71.43% (p=0.000 n=10)
ETHIsToValidContractAddress/log/hit-router-48             7.000 ± 0%   2.000 ± 0%  -71.43% (p=0.000 n=10)
ETHIsToValidContractAddress/tx/miss-48                   74.000 ± 0%   2.000 ± 0%  -97.30% (p=0.000 n=10)
ETHIsToValidContractAddress/tx/hit-last-whitelisted-48   74.000 ± 0%   2.000 ± 0%  -97.30% (p=0.000 n=10)
geomean                                                   22.76        2.000       -91.21%

2. closeStream timer

                             │    base     │              new              │
                             │   sec/op    │   sec/op     vs base          │
GCWithPendingStreamResets-48   7.826m ± 1%   7.873m ± 2%  ~ (p=0.481 n=10)

                             │       base       │                   new                    │
                             │ pending-B/stream │ pending-B/stream  vs base                │

3. HTX gzip reader reuse

                      │     base     │                 new                 │
                      │    sec/op    │   sec/op     vs base                │
HtxHandleWsMessage-48   27.14µ ± 10%   12.24µ ± 6%  -54.90% (p=0.000 n=10)

                      │     base      │                 new                  │
                      │     B/op      │     B/op      vs base                │
HtxHandleWsMessage-48   41.381Ki ± 0%   1.134Ki ± 0%  -97.26% (p=0.000 n=10)

                      │    base    │                new                 │
                      │ allocs/op  │ allocs/op   vs base                │
HtxHandleWsMessage-48   26.00 ± 0%   21.00 ± 0%  -19.23% (p=0.000 n=10)

4. p2p bufio pools

                   │     base     │                 new                  │
                   │    sec/op    │    sec/op     vs base                │
StreamRoundTrip-48   2849.0n ± 2%   798.2n ± 13%  -71.98% (p=0.000 n=10)

                   │     base     │                 new                  │
                   │     B/op     │     B/op      vs base                │
StreamRoundTrip-48   9.602Ki ± 0%   1.509Ki ± 0%  -84.29% (p=0.000 n=10)

                   │    base    │                new                 │
                   │ allocs/op  │ allocs/op   vs base                │
StreamRoundTrip-48   6.000 ± 0%   3.000 ± 0%  -50.00% (p=0.000 n=10)

5. Tx.Hash without fmt

          │    base     │                 new                  │
          │   sec/op    │    sec/op     vs base                │
TxHash-48   2.711µ ± 6%   1.081µ ± 24%  -60.12% (p=0.000 n=10)

          │    base    │                new                 │
          │    B/op    │    B/op     vs base                │
TxHash-48   536.0 ± 0%   400.0 ± 0%  -25.37% (p=0.000 n=10)

          │    base    │                new                 │
          │ allocs/op  │ allocs/op   vs base                │
TxHash-48   16.00 ± 0%   10.00 ± 0%  -37.50% (p=0.000 n=10)

Testing

  • go test -tags mocknet on all touched packages passes, as on the base commit: common (also without the tag), bifrost/pkg/chainclients/{evm,ethereum,shared/evm}, bifrost/observer, bifrost/oracle/..., bifrost/p2p/.... Also x/thorchain.
  • -race: bifrost/p2p, and the HTX test.
  • FuzzLowerHexAddress ran for 20 s.
  • golangci-lint v1.63.4 with the repo config on the touched packages reports only the pre-existing SA4006 in common/address_mocknet_test.go. gofumpt is clean. go build ./... passes.

Follow-ups (not done: can't be shown exactly equivalent)

Items 1–6 are now in !5231 (merged), one commit each, with the behavior differences pinned by tests. Ranked by CPU. Each states exactly what would change.

  1. RemoveSignedTxItem: check Has before Delete (~0.6–0.7 s, 1.3–1.4 %).
    • Today: every tx in every EVM block writes a leveldb delete marker for a signed-tx hash that was almost never stored (getTxInOptimized L385: 0.25 s). The unstuck loop's GetSignedTxItems then iterates through those markers (0.38 s).
    • What would change: if the key is missing and leveldb is in a write-failure or read-only state, the call returns nil instead of the write error, and that log line goes away.
  2. Keys.GetSignerInfo: cache the keyring record (~0.26 s, plus ~22 M objects, about 4.5 % of all allocations).
    • Today: every FetchNodeStatus (about once a second via sendDeck) re-decrypts the keyring entry (jose2go + PBKDF2).
    • What would change: a keyring file edited at runtime wouldn't be re-read, and a transient read error would no longer panic.
  3. Tron GetLatestBlock: decode only the block number (~0.2–0.3 s).
    • What would change: a wrongly typed non-number field would no longer fail the height lookup. That is the same trade as !5225 (merged), and FetchTxs still fully decodes the block.
  4. PruneBlockMeta: decode only height (~0.15–0.25 s).
    • What would change: a stored entry with a mistyped non-height field would no longer abort the prune. Bifrost itself only ever writes well-typed entries.
  5. UTXO: reuse the block already scanned for the coinbase value. This halves the multi-MB getblock RPCs on BTC, LTC, BCH, DOGE and ZEC.
    • What would change: it adds a cache, and it succeeds where a second fetch would have failed transiently.
  6. getTxInOptimized: a worker pool instead of a goroutine per receipt (~0.2 s of stack growth).
    • What would change: a restructure of the concurrency, and it overlaps !5224 (merged).

Other ideas worth a decision:

  • eth_getBlockReceipts for ETH, which today makes one receipt RPC per tx, serialized at concurrency 1.
  • Turning off RPC gzip for nodes on a local network (gzip is 2.7 % of CPU).
  • GOGC/GOMEMLIMIT tuning: the live heap is mostly pointer-free leveldb buffers.
  • Long-lived p2p streams instead of a stream per batch per peer (yamux costs 3.8 s; ~4,400 goroutines are created per second).
  • Mimir EVMDisableContractWhitelist read once per block instead of once per receipt.
  • MEXC: request only the needed symbols.

Not optimizations, found along the way (verified; the three bugs are fixed in !5224 (merged))

  • bifrost/pkg/chainclients/tron/api/tron_api.go:333: api.logger.Err(err) has no .Msg(), so this error is never logged. The error is still returned.
  • bifrost/pkg/chainclients/evm/evm_block_scanner.go:534: errors.Is(err, ethereum.NotFound) tests the outer err, which is always nil there, instead of elem.Error. On the batch-receipt path, NotFound receipts are always logged as errors.
  • getTxInOptimized blocks forever if Concurrency is 0 (unbuffered semaphore). Nothing validates it; every EVM config sets at least 1.
  • ETH extractTxs starts a goroutine for every tx before taking the semaphore, so ~200 sit parked per block at concurrency 1.

🤖 Generated with Claude Code

Edited by Son of Odin

Merge request reports

Loading
Loading