Skip to content

Provide sensible real-world benchmarks against CGo implementation

Current state of benchmarks vs CGo implementation of mattn:

  • only 2 benchmark functions are implemented: namely, simplest forms of insert and select (single INT column, no table index, no WHERE clauses). Those cases are not real-world, and we need something that would make sense.
  • currently we run benchmarks against tables of different sizes (commonly the magnitude of 10: 10,100,1000...). It has been shown already that performance factor stabilizes after reaching a sensible table size (say, 10 000 rows). Thus repeating benchmarks on different table sizes is useless, and instead we should stick to a table of sensible fixed size.
  • due to previous bullet, we use graphical plotting to better visualize benchmark results, but the pictures provide little info and mostly serve as a decoration. We should have benchmarks results that can be understood clearly from text output. We should avoid boilerplate plotting code in benchmarking package.

To address the above points, I refactored benchmarks in following ways:

  • the real-world benchmark scenarios were implemented. the logic is inspired-by and closely repeat tests described in https://www.sqlite.org/speed.html
  • the size of test table is now fixed at 100k rows, and we do not run benchmarks many times on different table sizes, since factors do not differ after reaching a sensible table size
  • the plotting code is cut-off and benchmark results now can be understood from textual representation

The new benchmarking aims to point us to the real bottleneck places of pure-Go implementation.

Output exmple:

=== RUN   Test_BenchmarkSQLite

goos:   darwin
goarch: amd64
cpu:    Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
repeat: 1 time(s)
in-memory SQLite: false

bench_create_index                  |  1.80x | CGo: 120.880 ms/op | Pure-Go: 217.574 ms/op
bench_select_on_string_comparison   |  2.25x | CGo:  19.326 ms/op | Pure-Go:  43.498 ms/op
bench_select_with_index             |  5.84x | CGo:   0.002 ms/op | Pure-Go:   0.014 ms/op
bench_select_without_index          |  1.50x | CGo:   6.071 ms/op | Pure-Go:   9.111 ms/op
bench_insert                        |  1.17x | CGo:   0.481 ms/op | Pure-Go:   0.565 ms/op
bench_insert_in_transaction         |  1.78x | CGo:   0.004 ms/op | Pure-Go:   0.006 ms/op
bench_insert_into_indexed           |  1.62x | CGo:   0.008 ms/op | Pure-Go:   0.013 ms/op
bench_insert_from_select            |  1.80x | CGo:  30.409 ms/op | Pure-Go:  54.703 ms/op
bench_update_text_with_index        |  3.26x | CGo:   0.004 ms/op | Pure-Go:   0.013 ms/op
bench_update_with_index             |  4.20x | CGo:   0.003 ms/op | Pure-Go:   0.011 ms/op
bench_update_without_index          |  1.40x | CGo:   6.421 ms/op | Pure-Go:   9.010 ms/op
bench_delete_without_index          |  1.28x | CGo: 180.734 ms/op | Pure-Go: 231.105 ms/op
bench_delete_with_index             |  1.85x | CGo:  34.284 ms/op | Pure-Go:  63.569 ms/op
--- PASS: Test_BenchmarkSQLite (171.62s)

PS: due to huge git diff, it makes more sense to look directly to the merge source branch: https://gitlab.com/glebarez/sqlite/-/tree/benchmarks2/benchmark

Edited by glebarez

Merge request reports

Loading