Depends on: %Pipelining - Part 3

Prerequisite for: -

Motivation

Objective: Reduce latency and sustain full load

Key Result: Memory usage of the mempool is bounded.

Overall goal of this milestone is to make the mempool safer and faster.

Scope

Safer mempool:

• The mempool currently only bounds the total number of valid manager operations. We will instead bound both the total number of valid operations of all kinds, and the total size (bytes) of all these operations.

Faster mempool:

• Remove redundant information maintained in the mempool since the pipelining improvements.
• Update the error classification of conflicting operations to avoid needlessly reclassifying these operations when the mempool switches heads.

Design

See details in issues.

Work Breakdown

Features (2-3w)

• #5196 (closed) Mempool: bound number and total size of valid operations of all kinds
  • adapt existing tests !8501 (merged)
  • code and new tests !6787 (merged)
• (1-2w) #5197 (closed) Mempool: indicate minimal fees for an operation to be accepted (ETA: early June)
  • indicate minimal fee when mempool is full
    • (days) plugin: add a function that computes minimal fee !8573 (merged)
    • (days) shell: indicate minimal fee in shell error !8640 (merged)
  • indicate minimal fee when there is a same-manager conflict
    • (days) shell: add an operation map indexed by manager !8699 (merged)
    • (days) plugin: add a function that computes minimal fee, taking replace-by-fee factor into account !8842 (merged)
    • (days) shell: indicate minimal fee in shell error !9016 (merged)
• (1-2d) #5198 (closed) Mempool: rework classification of conflict errors !9314 (merged) (ETA: mid June)

Misc Technical Debt (2-4w) #5279 (closed)

• (1-2d) #4448 (closed) Remove old instances of the term "precheck" (ETA: early June)
  • (days) in the prevalidator/mempool !8857 (merged)
  • (hours) in shell_metrics !9137 (merged) nomadic-labs/grafazos!38 (merged)
• (1-2d) #2065 (closed) known_valid mempool field should be a set instead of a list (ETA: early June)
• (1-2 weeks) #5541 (closed) Reinforce various mempool tests !9869 (merged) !9914 (merged) !9943 (merged)
• (1-2d) Deprecate obsolete disable_precheck from node config !8963 (merged)
• (1d) #5282 (closed) Mempool: rename Filter module to Plugin !9583 (merged)

Streamline and reinforce mempool implementation (2-4w) #5280

• design is currently unclear
• remove dead/redundant code (non-trivial to determine which parts are now dead or redundant)
• some files don't make much sense now that there is no longer a "legacy mempool" living alongside the "new mempool"
• update all tests
• risk: may need to add lots of tests

Maybe

• (1-8d) #5281 Mempool: try to initialize mempool for new protocol even if there was no mempool for the previous protocol
  • risk: may realize that tests are missing
• (3-8d) #1725 Prevalidator: on_launch should load filter configuration from a file