prevalidator: improve mempool advertisement priority
Context
This MR aims to reduce the number of operation re-advertise for each level.
An operation is advertised once it is classified, and each time it is re-classified in the prevalidator. If a valid operation living in the mempool is not included in a block, it will be reclassified and re-advertised in the next level.
In the %(2023Q4) - Layer 1 - Reduce Block time project, we monitored that a lot of time can be spent in the mempool, in both advertising and handling operations already advertised and received on previous level.
This MR improve the priority type in the pending data-structure to know if the operation has already been classified and therefore advertised, or if the operation is New and needs to be advertised. This improvement is used to avoid re-advertising non-consensus operation already advertised.
This change reduce the number of operation send to peers after each flush when the mempool is loaded (operation in block < operations in mempool). On our experiment we went from ~140k received operation to ~13.5k.
Teztale report without this MR changes:
Mempool profiling report:
on_notify .................................................................... 1126 517.634ms 100%
already handled ............................................................ 0
classified in mempool .................................................... 0
endorsement ............................................................ 9165
other .................................................................. 108927
preendorsement ......................................................... 18183
may fetch operation ........................................................ 138407
not already handled ........................................................ 471
Teztale report with this MR changes:
Mempool profiling report:
on_notify .................................................................... 351 134.493ms 100%
already handled ............................................................ 0
classified in mempool .................................................... 0
endorsement ............................................................ 1679
other .................................................................. 718
preendorsement ......................................................... 378
is already handled ......................................................... 0
is_pending ............................................................... 2
may fetch operation ........................................................ 13573
not already handled ........................................................ 350
Manually testing the MR
To test the changes in the priority type from the pending data-structure, run:
dune exec src/lib_shell/test/main.exe -- --file test_prevalidator_pending_operations.ml
To run newly added tests run:
dune exec src/lib_shell/test/main.exe -- --file test_prevalidator.ml -v
Checklist
-
Document the interface of any function added or modified (see the coding guidelines) -
Document any change to the user interface, including configuration parameters (see node configuration) -
Provide automatic testing (see the testing guide). -
For new features and bug fixes, add an item in the appropriate changelog ( docs/protocols/alpha.rstfor the protocol and the environment,CHANGES.rstat the root of the repository for everything else). -
Select suitable reviewers using the Reviewersfield below. -
Select as Assigneethe next person who should take action on that MR

