feat: Integration with persy using background ops
Hi,
This is a rework of the Integration with Persy and replace !107 (closed), depending on the last improvements in the master branch of Persy regarding background fsync operations, that got quite faster and more stable, this allowed me to simplify a lot the integration removing the need of a write cache, now the operation when possible are 1 to 1 to Persy operations, in terms of durability of this integration is the same of Normal of sqlite (means not data loss on program crash, potential loss of recent transactions on OS crash).
Regarding performances I ran the rjbench 15 times against both Persy and Sqlite here a short version of the results:
Persy
1st run:
real 0m19.621s
user 0m0.407s
sys 0m0.112s
15th run:
real 0m30.735s
user 0m0.497s
sys 0m0.131s
Sqlite
1st run:
real 0m21.644s
user 0m0.315s
sys 0m0.190s
15th run:
real 0m25.904s
user 0m0.399s
sys 0m0.146s
From this number the performance are comparable, Persy seems a bit slower on the long run but does a bit more early cleaning of journal that do not produce big pause, as well I'm in continue toning of Persy right now.
In attachment the full log of the run
I also ran the rjbench against persy 100 times just to see the speed drop with data grow, this is the cleaned log of the 100 runs : speed_100.log
I checked the database size after this runs and was around 750 Mbs the RAM used at the end was ~ 1.5GB with the database cache configured at 250MB
In terms of release of the relative Persy improvements I'm expecting to release them around January with the 1.2 release.
That's all for now.
-
I agree to release my code and all other changes of this MR under the Apache-2.0 license