Integration with Persy
This provide the integration wit persy with the last public release, it can be used as the base for stability and performance testing.
The current implementation have the integration with persy that is pretty straight forward and an additional write cache to reduce the latency of write operations batching multiple puts in a single transaction.
In the specific of the write cache it has now a buffer of 1000 operations that stay in memory for max 1 second, this parameters are configurable but in the current implementation are not yet forwarded on the conduit Config structure, as well the current implementation is specialized on the persy backend but can be easily generalized to support any other backend that have the same write latency issues.
for the timed flush i added the timer dependency, if there is a better way to execute scheduled operations in conduit, let me know i will refactor accordingly.
About performance the current implementation take ~1M and 20 secs to run the Romeo & Juliet benchmark and take ~13Mb of disk usage.
In terms of memory usage i did multiple run of the Romeo & Juliet benchmark having the Persy cache configured at 32Mb and this are the results:
Run Memory Resident Memory
1th 21 41
2th 30 48
3th 41 59
4th 43 60
5th 44 52
6th 48 67
7th 49 68
8th 50 69
9th 51 69
10th 54 73
11th 55 74
12th 60 78
13th 63 82
14th 64 83
15th 65 84
16th 65 84
16th 69 87
17th 70 88(data in MBs)
I see the memory keep growing a bit also at the end, not sure where this additional memory usage come from anyway the amount look trivial, and may even be related to the memory management itself.
when I did this run I didn't monitor the performance of each run, but the last run took quite long time I think around ~7/8 minutes, I saw that the database grow to 223 MB so the additional time may depends on the additional data, anyway I thought this is interesting to report.
I did some profiling after I did all the previous run, and I noticed most of the cost was actually around reading data, so toning the cache size depending on the use cases should help on the reading performance, I noticed that there are some optiomization on the Persy side that I will do in future releases.
In terms of Persy releases when I started this I was working on the documentation for releasing the 1.0, that is complete on the coding side, as soon that release is out I will come back to this and update this code with the few breaking changes that the 1.0 will have.
- [ x ] I agree to release my code and all other changes of this MR under the Apache-2.0 license
Regards