Sharing of extender segments in memory
The statistics of extender segments of the blockchain state around 2019-10-16, stored using the "aggressive" path encoding at the end.
There are lots of extender segments with the same contents under size 16 bits. This is because the "aggressive" path encoding which use shorten path names to 16 bit hashes.
This statistics should change in other path encodings, but it should be also true that many segments can be sharable.
- Check the segment statistics of "identity-hex" path encoding.
- Assess hashcons strategy: cont and gain.
- Implement hashcons of segments.
last root index= 734365438
extenders=2303490
--- extender statistics
length, distinct, total, dupe_ratio
0, 0, 0, nan
1, 2, 403635, 201817.50
2, 4, 55895, 13973.75
3, 8, 10803, 1350.38
4, 16, 4252, 265.75
5, 32, 3291, 102.84
6, 64, 1031, 16.11
7, 126, 574, 4.56
8, 169, 1538, 9.10
9, 130, 54113, 416.25
10, 76, 4192, 55.16
11, 45, 11754, 261.20
12, 33, 654985, 19848.03
13, 20, 88194, 4409.70
14, 13, 285486, 21960.46
15, 8, 290261, 36282.62
16, 6, 72230, 12038.33
17, 3, 3, 1.00
18, 1, 1, 1.00
19, 1, 43, 43.00
20, 0, 0, nan
21, 0, 0, nan
22, 1, 1, 1.00
23, 0, 0, nan
24, 0, 0, nan
25, 0, 0, nan
26, 0, 0, nan
27, 0, 0, nan
28, 0, 0, nan
29, 0, 0, nan
30, 0, 0, nan
31, 0, 0, nan
32, 0, 0, nan
33, 0, 0, nan
34, 570, 583, 1.02
35, 0, 0, nan
36, 0, 0, nan
37, 0, 0, nan
38, 0, 0, nan
39, 0, 0, nan
40, 0, 0, nan
41, 0, 0, nan
42, 0, 0, nan
43, 0, 0, nan
44, 0, 0, nan
45, 0, 0, nan
46, 0, 0, nan
47, 0, 0, nan
48, 0, 0, nan
49, 0, 0, nan
50, 0, 0, nan
51, 0, 0, nan
52, 0, 0, nan
53, 0, 0, nan
54, 0, 0, nan
55, 0, 0, nan
56, 0, 0, nan
57, 0, 0, nan
58, 0, 0, nan
59, 0, 0, nan
60, 0, 0, nan
61, 0, 0, nan
62, 0, 0, nan
63, 0, 0, nan
64, 0, 0, nan
65, 0, 0, nan
66, 0, 0, nan
67, 0, 0, nan
68, 0, 0, nan
69, 0, 0, nan
70, 0, 0, nan
71, 0, 0, nan
72, 0, 0, nan
73, 0, 0, nan
74, 0, 0, nan
75, 0, 0, nan
76, 0, 0, nan
77, 0, 0, nan
78, 0, 0, nan
79, 0, 0, nan
80, 0, 0, nan
81, 0, 0, nan
82, 0, 0, nan
83, 0, 0, nan
84, 0, 0, nan
85, 0, 0, nan
86, 0, 0, nan
87, 0, 0, nan
88, 0, 0, nan
89, 0, 0, nan
90, 0, 0, nan
91, 0, 0, nan
92, 0, 0, nan
93, 0, 0, nan
94, 0, 0, nan
95, 0, 0, nan
96, 0, 0, nan
97, 0, 0, nan
98, 0, 0, nan
99, 0, 0, nan
100, 0, 0, nan
101, 0, 0, nan
102, 0, 0, nan
103, 0, 0, nan
104, 0, 0, nan
105, 0, 0, nan
106, 0, 0, nan
107, 0, 0, nan
108, 0, 0, nan
109, 0, 0, nan
110, 0, 0, nan
111, 0, 0, nan
112, 0, 0, nan
113, 0, 0, nan
114, 0, 0, nan
115, 0, 0, nan
116, 0, 0, nan
117, 0, 0, nan
118, 0, 0, nan
119, 0, 0, nan
120, 0, 0, nan
121, 0, 0, nan
122, 0, 0, nan
123, 4, 4, 1.00
124, 2, 2, 1.00
125, 2, 2, 1.00
126, 10, 10, 1.00
127, 10, 10, 1.00
128, 18, 18, 1.00
129, 44, 44, 1.00
130, 58, 58, 1.00
131, 197, 197, 1.00
132, 320, 320, 1.00
133, 602, 620, 1.03
134, 1283, 1283, 1.00
135, 2514, 2515, 1.00
136, 4936, 4936, 1.00
137, 9720, 9724, 1.00
138, 18213, 18213, 1.00
139, 33721, 33722, 1.00
140, 54415, 54420, 1.00
141, 71628, 71637, 1.00
142, 65081, 65085, 1.00
143, 28689, 28713, 1.00
144, 4761, 4762, 1.00
145, 2323, 2372, 1.02
146, 3116, 3150, 1.01
147, 2762, 2849, 1.03
148, 1881, 2277, 1.21
149, 1809, 2183, 1.21
150, 2740, 3372, 1.23
151, 4198, 5026, 1.20
152, 6209, 6891, 1.11
153, 6957, 7195, 1.03
154, 4862, 5063, 1.04
155, 2306, 2425, 1.05
156, 2240, 2286, 1.02
157, 2913, 2928, 1.01
158, 3179, 3195, 1.01
159, 3195, 3195, 1.00
160, 2818, 2818, 1.00
161, 2228, 2228, 1.00
162, 1373, 1373, 1.00
163, 783, 783, 1.00
164, 477, 477, 1.00
165, 304, 304, 1.00
166, 271, 271, 1.00
167, 239, 239, 1.00
168, 32, 32, 1.00
169, 0, 0, nan
170, 0, 0, nan
171, 0, 0, nan
172, 0, 0, nan
173, 0, 0, nan
174, 0, 0, nan
175, 0, 0, nan
176, 815, 815, 1.00
177, 0, 0, nan
178, 0, 0, nan
179, 0, 0, nan
180, 0, 0, nan
181, 0, 0, nan
182, 0, 0, nan
183, 0, 0, nan
184, 0, 0, nan
185, 0, 0, nan
186, 0, 0, nan
187, 0, 0, nan
188, 0, 0, nan
189, 0, 0, nan
190, 0, 0, nan
191, 0, 0, nan
192, 0, 0, nan
193, 0, 0, nan
194, 0, 0, nan
195, 0, 0, nan
196, 0, 0, nan
197, 0, 0, nan
198, 0, 0, nan
199, 0, 0, nan
200, 0, 0, nan
201, 0, 0, nan
202, 0, 0, nan
203, 0, 0, nan
204, 2, 2, 1.00
205, 8, 8, 1.00
206, 0, 0, nan
207, 10, 10, 1.00
208, 17, 17, 1.00
209, 31, 31, 1.00
210, 59, 59, 1.00
211, 101, 101, 1.00
212, 130, 130, 1.00
213, 120, 120, 1.00
214, 61, 61, 1.00
215, 5, 5, 1.00
216, 0, 0, nan
217, 9, 10, 1.11
218, 10, 11, 1.10
219, 8, 11, 1.38
220, 1, 1, 1.00
221, 2, 2, 1.00
222, 3, 4, 1.33
Edited by Jun Furuse