Provide sensible real-world benchmarks against CGo implementation
Current state of benchmarks vs CGo implementation of mattn:
- only 2 benchmark functions are implemented: namely, simplest forms of insert and select (single INT column, no table index, no WHERE clauses). Those cases are not real-world, and we need something that would make sense.
- currently we run benchmarks against tables of different sizes (commonly the magnitude of 10: 10,100,1000...). It has been shown already that performance factor stabilizes after reaching a sensible table size (say, 10 000 rows). Thus repeating benchmarks on different table sizes is useless, and instead we should stick to a table of sensible fixed size.
- due to previous bullet, we use graphical plotting to better visualize benchmark results, but the pictures provide little info and mostly serve as a decoration. We should have benchmarks results that can be understood clearly from text output. We should avoid boilerplate plotting code in benchmarking package.
To address the above points, I refactored benchmarks in following ways:
- the real-world benchmark scenarios were implemented. the logic is inspired-by and closely repeat tests described in https://www.sqlite.org/speed.html
- the size of test table is now fixed at 100k rows, and we do not run benchmarks many times on different table sizes, since factors do not differ after reaching a sensible table size
- the plotting code is cut-off and benchmark results now can be understood from textual representation
The new benchmarking aims to point us to the real bottleneck places of pure-Go implementation.
Output exmple:
=== RUN Test_BenchmarkSQLite
goos: darwin
goarch: amd64
cpu: Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
repeat: 1 time(s)
in-memory SQLite: false
bench_create_index | 1.80x | CGo: 120.880 ms/op | Pure-Go: 217.574 ms/op
bench_select_on_string_comparison | 2.25x | CGo: 19.326 ms/op | Pure-Go: 43.498 ms/op
bench_select_with_index | 5.84x | CGo: 0.002 ms/op | Pure-Go: 0.014 ms/op
bench_select_without_index | 1.50x | CGo: 6.071 ms/op | Pure-Go: 9.111 ms/op
bench_insert | 1.17x | CGo: 0.481 ms/op | Pure-Go: 0.565 ms/op
bench_insert_in_transaction | 1.78x | CGo: 0.004 ms/op | Pure-Go: 0.006 ms/op
bench_insert_into_indexed | 1.62x | CGo: 0.008 ms/op | Pure-Go: 0.013 ms/op
bench_insert_from_select | 1.80x | CGo: 30.409 ms/op | Pure-Go: 54.703 ms/op
bench_update_text_with_index | 3.26x | CGo: 0.004 ms/op | Pure-Go: 0.013 ms/op
bench_update_with_index | 4.20x | CGo: 0.003 ms/op | Pure-Go: 0.011 ms/op
bench_update_without_index | 1.40x | CGo: 6.421 ms/op | Pure-Go: 9.010 ms/op
bench_delete_without_index | 1.28x | CGo: 180.734 ms/op | Pure-Go: 231.105 ms/op
bench_delete_with_index | 1.85x | CGo: 34.284 ms/op | Pure-Go: 63.569 ms/op
--- PASS: Test_BenchmarkSQLite (171.62s)
PS: due to huge git diff, it makes more sense to look directly to the merge source branch: https://gitlab.com/glebarez/sqlite/-/tree/benchmarks2/benchmark
Edited by glebarez