Skip to content

Draft: Add rust read alternative with bindings

Sylvester Chin requested to merge sc1-rust-binding into master

This MR adds rust bindings for the FastMmapedFileRs class and its singleton method to_rust_metrics. Still a WIP since the to_rust_metrics's method signature differs from FastMmapedFile.to_metrics.

But this MR shows that we can have both C and Rust bindings co-exist in a gem. Referred to https://github.com/matsadler/magnus heavily on how to add rust bindings.

Set up

./bin/setup

You may need to clear the tmp folder and lib/*.bundle files to resolve a circular dependency issue.

Circular dependency detected: TOP => compile => compile:arm64-darwin22 => compile:fast_mmaped_file:arm64-darwin22 => copy:fast_mmaped_file:arm64-darwin22:3.0.5 => tmp/arm64-darwin22/stage/lib/fast_mmaped_file_rs.bundle => lib/fast_mmaped_file_rs.bundle => copy:fast_mmaped_file_rs:arm64-darwin22:3.0.5 => tmp/arm64-darwin22/stage/lib/fast_mmaped_file.bundle => copy:fast_mmaped_file:arm64-darwin22:3.0.5
/Users/sylvesterchin/.asdf/installs/ruby/3.0.5/bin/bundle:25:in `load'
/Users/sylvesterchin/.asdf/installs/ruby/3.0.5/bin/bundle:25:in `<main>'
Tasks: TOP => compile => compile:arm64-darwin22 => compile:fast_mmaped_file:arm64-darwin22 => copy:fast_mmaped_file:arm64-darwin22:3.0.5 => tmp/arm64-darwin22/stage/lib/fast_mmaped_file_rs.bundle => lib/fast_mmaped_file_rs.bundle => copy:fast_mmaped_file_rs:arm64-darwin22:3.0.5 => tmp/arm64-darwin22/stage/lib/fast_mmaped_file.bundle

Run benchmarks

Download the file in !79 (comment 1299524241) and run bundle exec bin/benchmark. Example output:

Warming up --------------------------------------
                   C     1.000  i/100ms
                rust     1.000  i/100ms
Calculating -------------------------------------
                   C     14.240  (± 0.0%) i/s -     72.000  in   5.060848s
                rust     14.086  (± 0.0%) i/s -     71.000  in   5.045187s

Comparison:
                   C:       14.2 i/s
                rust:       14.1 i/s - 1.01x  slower

The benchmark don't look pretty (the rust code is much slower than the existing C implementation), likely due to the usual suspects (unidiomatic Rust, wasteful string creation etc, wasting cpu cycles allocating & freeing mem). I removed concurrency to keep the base implementation simple for optimisation. We can add it back later (see schin1/mmap_parser_poc@d314325c).

Note: we are actually doing more work in Rust (opening the directory and parsing the file names, effectively line 30-32 https://gitlab.com/gitlab-org/ruby/gems/prometheus-client-mmap/-/blob/master/lib/prometheus/client/formats/text.rb#L30) compared to Prometheus::Client::Formats::Text.marshal_multiprocess(dir) which does some initial work in ruby.

Plenty of TODOs

  • fix spec
  • add rust tests
  • match C implementation's signatures
  • optimise to match C implementation performance (another branch)
Edited by Sylvester Chin

Merge request reports