Draft: Proto/baker: extend Baking.baking_minimal_timestamp to fix flaky tests

Context

Another alternative: !5341 (closed)

Fixes #3097 (closed).

When baking with minimal timestamp, in particular in tezt tests, we sometimes bake too quickly, and don't give the chance to operations notified by the node via [monitor_operations] to be received by the baker. This causes some random flakiness in tests.

As a fix, we force fetching mempool operations with {!pending_operations} RPC and add the retrieved appplied operations to the mempool received by the baker's operations worker.

Manually testing the MR

• See if we get less flaky tests with tezt:
• In the debug trace below of a flaky test:
  • A: an operation is injected
  • B: the mempool classified it and notifies it via monitor_operations
  • C: the client shows infos about the op
  • D: we are about to bake (with minimal timestamp)
  • E: we fetch the baker's operations pool: it has no manager operation (the notified op not received yet)
  • F: via pending_operations RPC, we get one manager operation
  • G: if we don't include the operation fetched in (F), the block is baked with no manager ops, and the test fails.

(A)
[18:12:32.859] [client1] ./tezos-client --endpoint http://localhost:16385 --base-dir /var/folders/b4/9mtbn8lx79v12xzmzlcrct_40000gn/T/tezt-54241/1/client1 --wait none originate sc rollup from tz1KqTpEZ7Yob7QbPE4Hy4Wo8fHG8LhKxZSx of kind arith booting with  --burn-cap 9999999
[18:12:33.074] [client1] Node is bootstrapped.
[18:12:33.077] [client1] Estimated gas: 1600.648 units (will add 100 for safety)
[18:12:33.077] [client1] Estimated storage: 6522 bytes added (will add 20 for safety)

(B)
[18:12:33.080] [node1] will notify: opNYPNQqpmxHVF2ndTMwepYx71QCZJhbLLamt857CQE9hGayiP2
[18:12:33.080] [node1] May 18 20:12:33.081 - prevalidator: injecting operation opNYPNQqpmxHVF2ndTMwepYx71QCZJhbLLamt857CQE9hGayiP2 (force:false)
[18:12:33.080] [node1] May 18 20:12:33.081 - prevalidator:   Request pushed on 2022-05-18T18:12:33.080-00:00, treated in 5us, completed in 132us 

(C)
[18:12:33.080] [client1] Operation successfully injected in the node.
[18:12:33.080] [client1] Operation hash is 'opNYPNQqpmxHVF2ndTMwepYx71QCZJhbLLamt857CQE9hGayiP2'
[18:12:33.080] [client1] NOT waiting for the operation to be included.
[18:12:33.080] [client1] Use command
[18:12:33.080] [client1]   tezos-client wait for opNYPNQqpmxHVF2ndTMwepYx71QCZJhbLLamt857CQE9hGayiP2 to be included --confirmations 1 --branch BLoAZoQPHsqyQYjwK2BcsZzTvNgbgwBex1TdkuvpDsrny5LGBcs
[18:12:33.080] [client1] and/or an external block explorer to make sure that it has been included.
[18:12:33.080] [client1] This sequence of operations was run:
[18:12:33.080] [client1]   Manager signed operations:
[18:12:33.080] [client1]     From: tz1KqTpEZ7Yob7QbPE4Hy4Wo8fHG8LhKxZSx
[18:12:33.080] [client1]     Fee to the baker: ꜩ0.000402
[18:12:33.080] [client1]     Expected counter: 4
[18:12:33.080] [client1]     Gas limit: 1701
[18:12:33.080] [client1]     Storage limit: 6542 bytes
[18:12:33.080] [client1]     Balance updates:
[18:12:33.080] [client1]       tz1KqTpEZ7Yob7QbPE4Hy4Wo8fHG8LhKxZSx ... -ꜩ0.000402
[18:12:33.081] [client1]       payload fees(the block proposer) ....... +ꜩ0.000402
[18:12:33.081] [client1]     Originate smart contract rollup of kind arith with boot sector ''
[18:12:33.081] [client1]       This smart contract rollup origination was successfully applied
[18:12:33.081] [client1]       Consumed gas: 1600.648
[18:12:33.081] [client1]       Storage size: 6522 bytes
[18:12:33.081] [client1]       Address: scr1FZyW9bghVdUCejpez6Bs2XzHLixe947Ud
[18:12:33.081] [client1]       Balance updates:
[18:12:33.081] [client1]         tz1KqTpEZ7Yob7QbPE4Hy4Wo8fHG8LhKxZSx ... -ꜩ1.6305
[18:12:33.081] [client1]         storage fees ........................... +ꜩ1.6305
[18:12:33.082] client1 exited with code 0.
[18:12:33.085] [sc-rollup-node4] ./tezos-sc-rollup-node-alpha config init on scr1FZyW9bghVdUCejpez6Bs2XzHLixe947Ud with operator tz1KqTpEZ7Yob7QbPE4Hy4Wo8fHG8LhKxZSx --data-dir /var/folders/b4/9mtbn8lx79v12xzmzlcrct_40000gn/T/tezt-54241/1/sc-rollup-node4 --rpc-addr 127.0.0.1 --rpc-port 16389
[18:12:33.193] [sc-rollup-node4] Smart-contract rollup node configuration written in /var/folders/b4/9mtbn8lx79v12xzmzlcrct_40000gn/T/tezt-54241/1/sc-rollup-node4/config.json
[18:12:33.193] sc-rollup-node4 exited with code 0.

(D)
[18:12:33.193] [client1] ./tezos-client --endpoint http://localhost:16385 --base-dir /var/folders/b4/9mtbn8lx79v12xzmzlcrct_40000gn/T/tezt-54241/1/client1 bake for bootstrap1 --minimal-timestamp
[18:12:33.399] [client1] # In bake
[18:12:33.406] [client1]  # In monitor_ops
[18:12:33.410] [client1] # minimal timestamp is: true
[18:12:33.410] [client1] # baking_minimal_timestamp

(E)
[18:12:33.410] [client1]  # nb manager ops is 0

(F)
[18:12:33.411] [client1]  # Now  manager ops is 1
[18:12:33.412] [client1] May 18 20:12:33.413 - alpha.baker.actions: Voting pass for liquidity baking toggle vote
[18:12:33.416] [node1] May 18 20:12:33.417 - validator.block: block at level 5 successfully pre-applied
[18:12:33.416] [node1] May 18 20:12:33.417 - validator.block: Request pushed on 2022-05-18T18:12:33.413-00:00, treated in 4us, completed in 3.511ms 
[18:12:33.419] [node1] May 18 20:12:33.420 - validator.block: block BLwNqEmnutAqYA3dGLpbe4iSP4wrHsaGVwaduQJFVGxJrFXZyHd successfully validated
[18:12:33.419] [node1] May 18 20:12:33.420 - validator.block: Request pushed on 2022-05-18T18:12:33.418-00:00, treated in 3us, completed in 1.131ms 
[18:12:33.420] [client1] May 18 20:12:33.421 - alpha.baker.actions: block BLwNqEmnutAqYA3dGLpbe4iSP4wrHsaGVwaduQJFVGxJrFXZyHd at level 5, round 0
[18:12:33.420] [client1] May 18 20:12:33.421 - alpha.baker.actions:   injected for delegate bootstrap1 (tz1KqTpEZ7Yob7QbPE4Hy4Wo8fHG8LhKxZSx)

(G)
[18:12:33.420] [client1] # Out bake

Checklist

• [-] Document the interface of any function added or modified (see the coding guidelines)
• [-] Document any change to the user interface, including configuration parameters (see node configuration)
• [-] Provide automatic testing (see the testing guide).
• [-] For new features and bug fixes, add an item in the appropriate changelog (docs/protocols/alpha.rst for the protocol and the environment, CHANGES.rst at the root of the repository for everything else).
• Select suitable reviewers using the Reviewers field below.
• Select as Assignee the next person who should take action on that MR