Skip to content

Improve cemented metadata accesses

Context

This MR adds a file descriptor cache for the zipped cemented metadata. On average, this improves the RPC performances of /chains/main/blocks/<cemented_block> by 6x in binary and by 2x in JSON. The next bottleneck, according to perf reports, seems to be the serialization.

Performances logs #### JSON WITHOUT patch 2023/06/05 14:26:52.792446 INFO Processed 256 blocks in 10.16s (16 calls, height 3178633) 2023/06/05 14:27:03.358618 INFO Processed 256 blocks in 10.56s (16 calls, height 3178889) 2023/06/05 14:27:13.834030 INFO Processed 256 blocks in 10.47s (16 calls, height 3179145) 2023/06/05 14:27:24.026723 INFO Processed 256 blocks in 10.19s (16 calls, height 3179401) 2023/06/05 14:27:34.114833 INFO Processed 256 blocks in 10.08s (16 calls, height 3179657) 2023/06/05 14:27:44.652529 INFO Processed 256 blocks in 10.53s (16 calls, height 3179913) 2023/06/05 14:27:54.806474 INFO Processed 240 blocks in 10.15s (15 calls, height 3180153) 2023/06/05 14:28:04.963310 INFO Processed 240 blocks in 10.15s (15 calls, height 3180393) 2023/06/05 14:28:15.517086 INFO Processed 240 blocks in 10.55s (15 calls, height 3180633) --------------- #### JSON WITH patch 2023/06/05 14:23:15.185349 INFO Processed 512 blocks in 10.19s (32 calls, height 3179385) 2023/06/05 14:23:25.216180 INFO Processed 496 blocks in 10.03s (31 calls, height 3179881) 2023/06/05 14:23:35.704061 INFO Processed 448 blocks in 10.48s (28 calls, height 3180329) 2023/06/05 14:23:46.018340 INFO Processed 416 blocks in 10.31s (26 calls, height 3180745) 2023/06/05 14:23:56.021193 INFO Processed 448 blocks in 10s (28 calls, height 3181193) 2023/06/05 14:24:06.382782 INFO Processed 496 blocks in 10.36s (31 calls, height 3181689) 2023/06/05 14:24:16.642140 INFO Processed 272 blocks in 10.25s (17 calls, height 3181961) 2023/06/05 14:24:26.802347 INFO Processed 448 blocks in 10.16s (28 calls, height 3182409) 2023/06/05 14:24:37.153619 INFO Processed 400 blocks in 10.35s (25 calls, height 3182809) 2023/06/05 14:24:47.321517 INFO Processed 288 blocks in 10.16s (18 calls, height 3183097) 2023/06/05 14:24:57.506351 INFO Processed 432 blocks in 10.18s (27 calls, height 3183529) 2023/06/05 14:25:07.672201 INFO Processed 400 blocks in 10.16s (25 calls, height 3183929) ================ #### Binary WITHOUT patch 2023/06/05 14:28:58.820203 INFO Processed 480 blocks in 10.06s (30 calls, height 3178857) 2023/06/05 14:29:09.103466 INFO Processed 496 blocks in 10.28s (31 calls, height 3179353) 2023/06/05 14:29:19.137784 INFO Processed 480 blocks in 10.03s (30 calls, height 3179833) 2023/06/05 14:29:29.484993 INFO Processed 480 blocks in 10.34s (30 calls, height 3180313) 2023/06/05 14:29:39.800624 INFO Processed 448 blocks in 10.31s (28 calls, height 3180761) 2023/06/05 14:29:50.138320 INFO Processed 448 blocks in 10.33s (28 calls, height 3181209) 2023/06/05 14:30:00.311949 INFO Processed 464 blocks in 10.17s (29 calls, height 3181673) 2023/06/05 14:30:10.422699 INFO Processed 432 blocks in 10.11s (27 calls, height 3182105) 2023/06/05 14:30:20.439164 INFO Processed 448 blocks in 10.01s (28 calls, height 3182553) --------------- #### Binary WITH patch 2023/06/05 14:33:00.639420 INFO Processed 3200 blocks in 10.02s (200 calls, height 3181577) 2023/06/05 14:33:10.670020 INFO Processed 2256 blocks in 10.03s (141 calls, height 3183833) 2023/06/05 14:33:20.670913 INFO Processed 2608 blocks in 10s (163 calls, height 3186441) 2023/06/05 14:33:30.723513 INFO Processed 2288 blocks in 10.05s (143 calls, height 3188729) 2023/06/05 14:33:40.829211 INFO Processed 2560 blocks in 10.1s (160 calls, height 3191289) 2023/06/05 14:33:50.876343 INFO Processed 2272 blocks in 10.04s (142 calls, height 3193561) 2023/06/05 14:34:00.912672 INFO Processed 1888 blocks in 10.03s (118 calls, height 3195449) 2023/06/05 14:34:10.938883 INFO Processed 2496 blocks in 10.02s (156 calls, height 3197945) 2023/06/05 14:34:20.941044 INFO Processed 2496 blocks in 10s (156 calls, height 3200441) 2023/06/05 14:34:30.974978 INFO Processed 2352 blocks in 10.03s (147 calls, height 3202793) 2023/06/05 14:34:41.035733 INFO Processed 2928 blocks in 10.06s (183 calls, height 3205721)

Checklist

  • Document the interface of any function added or modified (see the coding guidelines)
  • Document any change to the user interface, including configuration parameters (see node configuration)
  • Provide automatic testing (see the testing guide).
  • For new features and bug fixes, add an item in the appropriate changelog (docs/protocols/alpha.rst for the protocol and the environment, CHANGES.rst at the root of the repository for everything else).
  • Select suitable reviewers using the Reviewers field below.
  • Select as Assignee the next person who should take action on that MR
Edited by vbot

Merge request reports