undefined | Better HN

0 pointstoast011mo ago0 comments

libuv is more or less abstraction around an event loop for async i/o right?

The BEAM is also more or less an abstraction around an event loop for async i/o. If you want async i/o in nifs, I think you want to integrate with BEAM's event loop. Inside NIFs, I think you want to use enif_select [1] (and friends), available since OTP 20 originally from 2017-06-21. In a port driver, you'd use driver_select [2] which I think has been around forever --- there's mentions of changes in R13 which I think was mostly release 2009-11-20 (that may have been R13B though).

[1] https://www.erlang.org/doc/apps/erts/erl_nif.html#enif_selec...

[2] https://www.erlang.org/doc/apps/erts/erl_driver.html#driver_...

0 comments

hinkley11mo ago

It uses different threads in order to make infrequent blocking calls look like they are asynchronous.

When we (or at least some quantity of “we”) want is infrequent native calls to be able to fail without taking the BEAM down.

The problem with doing it with threads though is that a bad thread can still vomit all over working memory, still causing a panic even if it itself doesn’t panic.

toast0OP11mo ago

Oh I see. If you need to isolate your native code from your BEAM, then you've got several options.

a) run the native code as a separate process; either a port program, a c-node, or just a regular program that you interact with via some interprocess communication (sockets, pipes, signals, a shared filesystem, shared memory segments if you're brave)

b) some sort of sandybox thing; like compile to wasm and then jit back to (hopefully) safe native.

c) just run the native code, it's probably fine, hopefully. My experience with NIFs is that they are usually very short code that should be easy to review, so this isn't as bad as it sounds...

If your native code is short, option c is probably fine; if your native code is long, option a makes more sense. If you want to make things harder without real justification, b sounds good :P

hinkley11mo ago

I suspect the confusion of priorities comes from running native code that’s time consuming and wanting it to horizontally scale with your cluster, which is easier if you try to let the beam do it.

While what is easier for those of us not working on the beam is to put the glitchy code into its own service and put up with the maintenance overhead.

But when you have one or two solutions the friction to move to three becomes difficult. People start talking about how having dozens will be chaos. While true, sometimes you really do just need three and it’s not a slippery slope.

1 more reply

j / k navigate · click thread line to collapse

0 pointstoast011mo ago0 comments

libuv is more or less abstraction around an event loop for async i/o right?

[1] https://www.erlang.org/doc/apps/erts/erl_nif.html#enif_selec...

[2] https://www.erlang.org/doc/apps/erts/erl_driver.html#driver_...

0 comments

hinkley11mo ago

It uses different threads in order to make infrequent blocking calls look like they are asynchronous.

When we (or at least some quantity of “we”) want is infrequent native calls to be able to fail without taking the BEAM down.

The problem with doing it with threads though is that a bad thread can still vomit all over working memory, still causing a panic even if it itself doesn’t panic.

toast0OP11mo ago

Oh I see. If you need to isolate your native code from your BEAM, then you've got several options.

b) some sort of sandybox thing; like compile to wasm and then jit back to (hopefully) safe native.

c) just run the native code, it's probably fine, hopefully. My experience with NIFs is that they are usually very short code that should be easy to review, so this isn't as bad as it sounds...

If your native code is short, option c is probably fine; if your native code is long, option a makes more sense. If you want to make things harder without real justification, b sounds good :P

hinkley11mo ago

I suspect the confusion of priorities comes from running native code that’s time consuming and wanting it to horizontally scale with your cluster, which is easier if you try to let the beam do it.

While what is easier for those of us not working on the beam is to put the glitchy code into its own service and put up with the maintenance overhead.

1 more reply

j / k navigate · click thread line to collapse