undefined | Better HN

0 pointstlb8y ago0 comments

I recently switched from zeromq to straight libuv sockets with jsonl (\n-separated json) payloads. Because I'm working inside a Node process, combining zmq's threading model with Node's threading model was a pain. Now, there's a single IO thread which is the same as the Javascript engine thread, and I can use uv_work to run CPU-intensive tasks on multiple cores.

0 comments

Matthias2478y ago

Do you not allow \n inside your JSON or encode your JSON as base64? If not you might have problems with disambiguating frame ends from line breaks inside frames.

A common way for framing is to prepend each frame with it's encoded length. That's easier, faster and less error-prone than searching for ASCII delimeters.

AgentME8y ago

If you do the JSON serialization yourself, there's no reason for newlines to be in the JSON. (Newlines within strings are encoded as \n.)

1 more reply

tlbOP8y ago

I'm generating the JSON, either with custom C++ marshaling routines or with JSON.stringify which doesn't include newlines unless you give it extra arguments. I believe that any valid JSON can be converted to a single line by changing any '\n' bytes to ' '. Literal '\n' bytes are not allowed inside strings, and outside strings any whitespace is equivalent.

justinsaccount8y ago

a newline is encoded as the two characters "\n" in json and would not be confused with a literal \n (aka \x0a) character

1 more reply

j / k navigate · click thread line to collapse

0 pointstlb8y ago0 comments

0 comments

Matthias2478y ago

Do you not allow \n inside your JSON or encode your JSON as base64? If not you might have problems with disambiguating frame ends from line breaks inside frames.

A common way for framing is to prepend each frame with it's encoded length. That's easier, faster and less error-prone than searching for ASCII delimeters.

AgentME8y ago

If you do the JSON serialization yourself, there's no reason for newlines to be in the JSON. (Newlines within strings are encoded as \n.)

1 more reply

tlbOP8y ago

justinsaccount8y ago

a newline is encoded as the two characters "\n" in json and would not be confused with a literal \n (aka \x0a) character

1 more reply

j / k navigate · click thread line to collapse