Improve cemented metadata accesses
Context
This MR adds a file descriptor cache for the zipped cemented metadata. On average, this improves the RPC performances of /chains/main/blocks/<cemented_block>
by 6x in binary and by 2x in JSON. The next bottleneck, according to perf
reports, seems to be the serialization.
Performances logs
#### JSON WITHOUT patch
2023/06/05 14:26:52.792446 INFO Processed 256 blocks in 10.16s (16 calls, height 3178633)
2023/06/05 14:27:03.358618 INFO Processed 256 blocks in 10.56s (16 calls, height 3178889)
2023/06/05 14:27:13.834030 INFO Processed 256 blocks in 10.47s (16 calls, height 3179145)
2023/06/05 14:27:24.026723 INFO Processed 256 blocks in 10.19s (16 calls, height 3179401)
2023/06/05 14:27:34.114833 INFO Processed 256 blocks in 10.08s (16 calls, height 3179657)
2023/06/05 14:27:44.652529 INFO Processed 256 blocks in 10.53s (16 calls, height 3179913)
2023/06/05 14:27:54.806474 INFO Processed 240 blocks in 10.15s (15 calls, height 3180153)
2023/06/05 14:28:04.963310 INFO Processed 240 blocks in 10.15s (15 calls, height 3180393)
2023/06/05 14:28:15.517086 INFO Processed 240 blocks in 10.55s (15 calls, height 3180633)
---------------
#### JSON WITH patch
2023/06/05 14:23:15.185349 INFO Processed 512 blocks in 10.19s (32 calls, height 3179385)
2023/06/05 14:23:25.216180 INFO Processed 496 blocks in 10.03s (31 calls, height 3179881)
2023/06/05 14:23:35.704061 INFO Processed 448 blocks in 10.48s (28 calls, height 3180329)
2023/06/05 14:23:46.018340 INFO Processed 416 blocks in 10.31s (26 calls, height 3180745)
2023/06/05 14:23:56.021193 INFO Processed 448 blocks in 10s (28 calls, height 3181193)
2023/06/05 14:24:06.382782 INFO Processed 496 blocks in 10.36s (31 calls, height 3181689)
2023/06/05 14:24:16.642140 INFO Processed 272 blocks in 10.25s (17 calls, height 3181961)
2023/06/05 14:24:26.802347 INFO Processed 448 blocks in 10.16s (28 calls, height 3182409)
2023/06/05 14:24:37.153619 INFO Processed 400 blocks in 10.35s (25 calls, height 3182809)
2023/06/05 14:24:47.321517 INFO Processed 288 blocks in 10.16s (18 calls, height 3183097)
2023/06/05 14:24:57.506351 INFO Processed 432 blocks in 10.18s (27 calls, height 3183529)
2023/06/05 14:25:07.672201 INFO Processed 400 blocks in 10.16s (25 calls, height 3183929)
================
#### Binary WITHOUT patch
2023/06/05 14:28:58.820203 INFO Processed 480 blocks in 10.06s (30 calls, height 3178857)
2023/06/05 14:29:09.103466 INFO Processed 496 blocks in 10.28s (31 calls, height 3179353)
2023/06/05 14:29:19.137784 INFO Processed 480 blocks in 10.03s (30 calls, height 3179833)
2023/06/05 14:29:29.484993 INFO Processed 480 blocks in 10.34s (30 calls, height 3180313)
2023/06/05 14:29:39.800624 INFO Processed 448 blocks in 10.31s (28 calls, height 3180761)
2023/06/05 14:29:50.138320 INFO Processed 448 blocks in 10.33s (28 calls, height 3181209)
2023/06/05 14:30:00.311949 INFO Processed 464 blocks in 10.17s (29 calls, height 3181673)
2023/06/05 14:30:10.422699 INFO Processed 432 blocks in 10.11s (27 calls, height 3182105)
2023/06/05 14:30:20.439164 INFO Processed 448 blocks in 10.01s (28 calls, height 3182553)
---------------
#### Binary WITH patch
2023/06/05 14:33:00.639420 INFO Processed 3200 blocks in 10.02s (200 calls, height 3181577)
2023/06/05 14:33:10.670020 INFO Processed 2256 blocks in 10.03s (141 calls, height 3183833)
2023/06/05 14:33:20.670913 INFO Processed 2608 blocks in 10s (163 calls, height 3186441)
2023/06/05 14:33:30.723513 INFO Processed 2288 blocks in 10.05s (143 calls, height 3188729)
2023/06/05 14:33:40.829211 INFO Processed 2560 blocks in 10.1s (160 calls, height 3191289)
2023/06/05 14:33:50.876343 INFO Processed 2272 blocks in 10.04s (142 calls, height 3193561)
2023/06/05 14:34:00.912672 INFO Processed 1888 blocks in 10.03s (118 calls, height 3195449)
2023/06/05 14:34:10.938883 INFO Processed 2496 blocks in 10.02s (156 calls, height 3197945)
2023/06/05 14:34:20.941044 INFO Processed 2496 blocks in 10s (156 calls, height 3200441)
2023/06/05 14:34:30.974978 INFO Processed 2352 blocks in 10.03s (147 calls, height 3202793)
2023/06/05 14:34:41.035733 INFO Processed 2928 blocks in 10.06s (183 calls, height 3205721)
Checklist
-
Document the interface of any function added or modified (see the coding guidelines) -
Document any change to the user interface, including configuration parameters (see node configuration) -
Provide automatic testing (see the testing guide). -
For new features and bug fixes, add an item in the appropriate changelog ( docs/protocols/alpha.rst
for the protocol and the environment,CHANGES.rst
at the root of the repository for everything else). -
Select suitable reviewers using the Reviewers
field below. -
Select as Assignee
the next person who should take action on that MR
Edited by vbot