[backport] Complete the BIP155 implementation and upgrade to TORv3

Summary

This is a backport of https://github.com/bitcoin/bitcoin/pull/19954

This also includes a partial backport of https://github.com/bitcoin/bitcoin/pull/20564

This PR contains the two remaining commits from #19031 to complete the BIP155 implementation:

net: CAddress & CAddrMan: (un)serialize as ADDRv2 net: advertise support for ADDRv2 via new message

plus one more commit:

tor: make a TORv3 hidden service instead of TORv2

Depends on !1475 (merged)

Test plan

• ninja all check check-functional-extended
• ./test/functional/test_runner.py --failfast -j $(nproc) $(echo {1..128} | sed 's/[^ ]*/p2p_addrv2_relay.py/g') to check for the fixed race condition in p2p_addrv2_relay
• Install a Tor instance and set ControlPort 9051, CookieAuthentication 1, CookieAuthFileGroupReadable 1 and depending on distro DataDirectoryGroupReadable 1 in /etc/tor/torrc. Add your user to the tor group: gpasswd -a tor user. Restart tor. You might need to log out and log in again to get your login session to have the additional group, check with id.
• bitcoind -datadir=<first datadir> -proxy=127.0.0.1:9050 -listen -debug=tor -debug=net to start the first instance. Check the log for tor: Got service ID v525rwi6aar2hpgmbryssymabh7ty4yzyjpym2pje5edhco23o6kofqd, advertising service v525rwi6aar2hpgmbryssymabh7ty4yzyjpym2pje5edhco23o6kofqd.onion:8333, this is your onion service address.
• bitcoind -datadir=<second datadir> -proxy=127.0.0.1:9050 -onlynet=onion -debug=tor -addnode=v525rwi6aar2hpgmbryssymabh7ty4yzyjpym2pje5edhco23o6kofqd.onion:8333 -rpcport=8442 to start the second instance.
• Check the logs to see if the second instance can connect to the first and they exchange sendaddrv2 after verack and then send addrv2.
• Do the same with -useextversion=1.

Some more useful commands to test this:

bitcoin-cli getpeerinfo > peerinfo.json

# Check that sendaddrv2 has been sent to all our peers
test $(jq '. | length' peerinfo.json) -eq $(jq '[.[] | select(.bytessent_per_msg.sendaddrv2 > 0)] | length' peerinfo.json) || echo failed

# List peers that have advertised via sendaddrv2 to us
jq '.[] | select(.bytesrecv_per_msg.sendaddrv2 > 0)' peerinfo.json

# List peers that have sent addrv2 to us
jq '.[] | select(.bytesrecv_per_msg.addrv2 > 0)' peerinfo.json

# List peers that we sent addrv2 to
jq '.[] | select(.bytesrecv_per_msg.addrv2 > 0)' peerinfo.json

# List onion peers
jq '.[] | select(.addr | contains("onion"))' peerinfo.json

I am going to break this down into more individual MRs. I put this up so others can take a look early. There currently should be a V3 onion that is sending / receiving addrv2 at v525rwi6aar2hpgmbryssymabh7ty4yzyjpym2pje5edhco23o6kofqd.onion:8333

I also just noticed that I didn't backport net: advertise support for ADDRv2 via new message correctly, the original commit meta data is lost. Will fix it soon.

Edit: same for scripted-diff: test: replace command with msgtype

Somehow I missed fixing the p2p_invalid_messages test. Will update it soon.

added 2 commits

• 24ec4bf8 - fix chrono
• b0a45b46 - test

Compare with previous version

Updated to address the std::chrono unit issues and make the p2p_invalid_messages test almost work. There is still unexpected stderr output, I'll have a closer look at it tomorrow.

I'll also make smaller MRs and add a proper test plan tomorrow.

resolved all threads

mentioned in merge request !1470 (merged)

added backport enhancement labels

added 55 commits

• b0a45b46...1bf4f39c - 22 commits from branch bitcoin-cash-node:master
• 9ed38baf - [backport] Remove two unneeded tests
• 5d395f4d - [backport] Move size limits to module-global
• b0a13b62 - [backport] Fix "invalid message size" test
• 66cc37bf - [backport] Refactor resource exhaustion test
• 213c0e08 - [backport] [tests] Don't import asyncio to test magic bytes
• 6d1f199f - [backport] Test buffered valid message
• 685deb84 - [backport] test: add p2p_invalid_messages logging
• fccf3b3f - [backport] test: refactor test_large_inv() into 3 tests with common method
• 136d0390 - [backport] test: improve msg sends and p2p disconnections in p2p_invalid_messages
• dafdc7e1 - [backport] test: hoist p2p values to test framework constants
• d1301b45 - [backport] scripted-diff: test: replace command with msgtype
• 464106a9 - [backport] scripted-diff: Rename mininode_lock to p2p_lock
• b26b1d0a - [backport] scripted-diff: Rename mininode to p2p
• 2632ae73 - [backport] [test] Remove final references to mininode
• e7b1a668 - [backport] test: Remove confusing cast to same type (int to int)
• 9368cb70 - [backport] test: Check that invalid peer traffic is accounted for
• 0d8111fd - [backport] [tools] add PoissonNextSend method that returns mockable time
• dd6fc3a2 - [backport] [tools] update nNextInvSend to use mockable time
• 803de345 - [backport] net: Make addr relay mockable
• fc41d41d - [backport] net: Pass connman const when relaying address
• b73b19af - [backport] test: Add basic addr relay test
• a3243de0 - [backport] scripted-diff: Replace strCommand with msg_type
• 6ccec616 - [backport] refactor: s/command/msg_type/ in CNetMsgMaker and CSerializedNetMsg
• c07bd638 - [backport] Drop CADDR_TIME_VERSION checks now that MIN_PEER_PROTO_VERSION is greater
• cf28c68e - [backport] [protocol] Remove unused CADDR_TIME_VERSION
• 9209776b - [backport] [protocol] Remove unused GETHEADERS_VERSION
• 9fe6d475 - [backport] Replace `hidden service` with `onion service`
• ead0534d - [backport] net: Log to net category for exceptions in ProcessMessages
• bb75158b - [backport] Add missing typeinfo includes
• 00effa02 - [backport] Support bypassing range check in ReadCompactSize
• 674ebab0 - [backport] net: CAddress & CAddrMan: (un)serialize as ADDRv2
• cdd60da9 - [backport] net: advertise support for ADDRv2 via new message
• fd84bb40 - [backport] tor: make a TORv3 hidden service instead of TORv2

Compare with previous version

added 2 commits

• a672fec8 - [backport] net: advertise support for ADDRv2 via new message
• 03d1b72d - [backport] tor: make a TORv3 hidden service instead of TORv2

Compare with previous version

This is almost ready, it still needs a better test plan.

I've seen this test failure once:

p2p_addrv2_relay.py failed, Duration: 5 s

stdout:
2022-01-16T10:32:39.719000Z TestFramework (INFO): Initializing test directory /home/ago/src/bitcoin-cash-node/build/test/tmp/bitcoin_test_runner_20220116_102955/p2p_addrv2_relay_417
2022-01-16T10:32:39.980000Z TestFramework (INFO): Create connection that sends addrv2 messages
2022-01-16T10:32:40.080000Z TestFramework (INFO): Send too-large addrv2 message
2022-01-16T10:32:40.135000Z TestFramework (INFO): Check that addrv2 message content is relayed and added to addrman
2022-01-16T10:32:40.339000Z TestFramework (ERROR): Assertion failed
Traceback (most recent call last):
  File "/home/ago/src/bitcoin-cash-node/test/functional/test_framework/test_framework.py", line 187, in main
    self.run_test()
  File "/home/ago/src/bitcoin-cash-node/test/functional/p2p_addrv2_relay.py", line 74, in run_test
    assert addr_receiver.addrv2_received_and_checked
AssertionError
2022-01-16T10:32:40.391000Z TestFramework (INFO): Stopping nodes
2022-01-16T10:32:45.553000Z TestFramework (WARNING): Not cleaning up dir /home/ago/src/bitcoin-cash-node/build/test/tmp/bitcoin_test_runner_20220116_102955/p2p_addrv2_relay_417
2022-01-16T10:32:45.553000Z TestFramework (ERROR): Test failed. Test logging available at /home/ago/src/bitcoin-cash-node/build/test/tmp/bitcoin_test_runner_20220116_102955/p2p_addrv2_relay_417/test_framework.log
2022-01-16T10:32:45.553000Z TestFramework (ERROR): Hint: Call /home/ago/src/bitcoin-cash-node/test/functional/combine_logs.py '/home/ago/src/bitcoin-cash-node/build/test/tmp/bitcoin_test_runner_20220116_102955/p2p_addrv2_relay_417' to consolidate all logs

stderr:

added 1 commit

• 9d890646 - test: Fix race condition in p2p_addrv2_relay.py test

Compare with previous version

changed the description

resolved all threads

changed the description

mentioned in issue #362 (closed)

changed the description