Create pipeline (like html-pipeline) in Rust gem

This would allow some of the transformations we do in the Banzai pipeline to be done in Rust.

Initially we might move some of the ruby transformations into the Ruby part of the gem. This could be a stepping stone into moving certain filters into Rust.

When doing this in Rust, we most likely would want to parse the HTML in the same way we use Nokogiri in Ruby, so that we can do more DOM like transformations. And this might be then converting back to HTML, and then reparsing with Nokogiri for any Ruby level transformations.

There is an interesting comment at https://github.com/kivikakk/comrak/pull/358#issuecomment-1902483657

which is going to be faster? Is it better to use something like kuchiki or maybe selma - convert the comrak HTML into another tree structure and operate on that and reconvert to HTML? Or take comrak's AST and either add information into the AST and pass to the standard HTML writer, or some combination of AST manipulation and a custom HTML writer?

It would indeed be faster to such things in Comrak; reparsing the HTML and constructing a new tree is (relatively) expensive. But I would heavily weigh up the complexity that would be added in service of such performance against the actual gains. Unless you get to a very high level of throughput, you will simply not notice the difference in speed or compute. You would notice the difference in a clunkier API, i.e. feeling like you're restricted by the fact that you want to do arbitrary DOM transformations without something that understands the DOM at all.

(imo, if that level of performance was truly a concern, you'd probably use something with streaming output, rather than building up an AST.)