Using Rust with Elixir for code reuse and performance (opens in new tab)

Heavy duty string bashing is a pessimal case for BEAM, yes. The two core data structures you could use for this, the binaries or the 'strings' (which are linked lists of characters), each have their own problems with this sort of algorithm. Erlang was designed to fling chunks around, maybe take a header apart before deciding where to fling a chunk around, not to grovel over every byte of some fairly-large string for detailed parsing.

Based on performance numbers I've seen, I'd expect a one-order-of-magnitude difference to be a more common case in general.

For me, Elixir with Rust NIFs seems like a match made in heaven.

The BEAM VM is amazing at many things but it is still a VM and while the BeamAsm Just In Time compiler added in 2020 can offer performance improvement gains of 130%, there are areas where the BEAM still won't outperform native code.

Erlang excels at network programming and binary data processing. Computation intensive math and heavy string processing are both good NIF use cases.

An article on string performance [1],pre-JIT, describes initial Elixir benchmark of 140s vs C's 3.74s. They managed to make a number of tradeoffs and get the Elixir benchmark improved to 13s. Part of how they did that was to not use unicode (IO.binstream instead of IO.stream), that alone gained them ~4s.

[1] https://blog.jola.dev/elixir-string-processing-optimization

Author here. I actually was not aware of cmark.ex - thanks for pointing it out.

In this case the code reuse was more important than pure native speed. We already had a Rust library that used pulldown-cmark [1] with some custom tweaks that we wanted to duplicate. Maybe this behavior could have been copied using cmark.ex too (we thought about doing this in pure Elixir, as mentioned in the post), but given how straightforward Rustler made integrating our existing code, this seems like the better choice.

[1] https://github.com/raphlinus/pulldown-cmark

Before Rust, it seemed there was a law that every sufficiently critical code path would eventually be rewritten in C or C++.

After Rust, giving up a bit of performance to not have to maintain the C code underneath seems preferred. And often you don't even sacrifice performance.

I’ve seen a major preference toward Rust for NIFs in Elixir land because of the guarantees. People like their BEAM guarantees and exiting the BEAM without strong guarantees feels risky.

Even if they did they may have preferred Rust because parsing arbitrary user provided input is one of the more risky things you can do in C. This is a perfect use case for Rust's memory safety model.

Author here. This is a good point. In this case we found it does slow down clean builds somewhat since we're pulling down + compiling Rust dependencies now as well as Elixir. While actually editing Elixir code I haven't experienced any noticeable slowdown for incremental builds - the Rust code is mostly static.

It's one reason I prefer lighter languages for NIF's. My personal preference is to use Nim and Nimler [1]. It generally compiles quicker than Rust while providing most of the same benefits. To be fair compiling a small Rust library doesn't take too much time. Especially compared to the performance numbers from those Rust NIFs!

1: https://github.com/wltsmrz/nimler

It’s also important to note that a NIF can bring down your whole BEAM.

It will be faaaar slower. If it is a good alternative depends on your need. Benchmark your situation