Draft: Add rust read alternative with bindings
This MR adds rust bindings for the FastMmapedFileRs
class and its singleton method to_rust_metrics
. Still a WIP since the to_rust_metrics
's method signature differs from FastMmapedFile.to_metrics
.
But this MR shows that we can have both C and Rust bindings co-exist in a gem. Referred to https://github.com/matsadler/magnus heavily on how to add rust bindings.
Set up
./bin/setup
You may need to clear the tmp
folder and lib/*.bundle
files to resolve a circular dependency issue.
Circular dependency detected: TOP => compile => compile:arm64-darwin22 => compile:fast_mmaped_file:arm64-darwin22 => copy:fast_mmaped_file:arm64-darwin22:3.0.5 => tmp/arm64-darwin22/stage/lib/fast_mmaped_file_rs.bundle => lib/fast_mmaped_file_rs.bundle => copy:fast_mmaped_file_rs:arm64-darwin22:3.0.5 => tmp/arm64-darwin22/stage/lib/fast_mmaped_file.bundle => copy:fast_mmaped_file:arm64-darwin22:3.0.5
/Users/sylvesterchin/.asdf/installs/ruby/3.0.5/bin/bundle:25:in `load'
/Users/sylvesterchin/.asdf/installs/ruby/3.0.5/bin/bundle:25:in `<main>'
Tasks: TOP => compile => compile:arm64-darwin22 => compile:fast_mmaped_file:arm64-darwin22 => copy:fast_mmaped_file:arm64-darwin22:3.0.5 => tmp/arm64-darwin22/stage/lib/fast_mmaped_file_rs.bundle => lib/fast_mmaped_file_rs.bundle => copy:fast_mmaped_file_rs:arm64-darwin22:3.0.5 => tmp/arm64-darwin22/stage/lib/fast_mmaped_file.bundle
Run benchmarks
Download the file in !79 (comment 1299524241) and run bundle exec bin/benchmark
. Example output:
Warming up --------------------------------------
C 1.000 i/100ms
rust 1.000 i/100ms
Calculating -------------------------------------
C 14.240 (± 0.0%) i/s - 72.000 in 5.060848s
rust 14.086 (± 0.0%) i/s - 71.000 in 5.045187s
Comparison:
C: 14.2 i/s
rust: 14.1 i/s - 1.01x slower
The benchmark don't look pretty (the rust code is much slower than the existing C implementation), likely due to the usual suspects (unidiomatic Rust, wasteful string creation etc, wasting cpu cycles allocating & freeing mem). I removed concurrency to keep the base implementation simple for optimisation. We can add it back later (see schin1/mmap_parser_poc@d314325c).
Note: we are actually doing more work in Rust (opening the directory and parsing the file names, effectively line 30-32 https://gitlab.com/gitlab-org/ruby/gems/prometheus-client-mmap/-/blob/master/lib/prometheus/client/formats/text.rb#L30) compared to Prometheus::Client::Formats::Text.marshal_multiprocess(dir)
which does some initial work in ruby.
Plenty of TODOs
-
fix spec -
add rust tests -
match C implementation's signatures -
optimise to match C implementation performance (another branch)