undefined | Better HN

0 pointsIanCal1y ago0 comments

Yes, it's not great. Space is annoying, though compression pretty much removes that as a concern (zstd is good for this, you can even have a custom dictionary). And yes, missing keys is annoying.

JSONL is handy, JSON that's in the form {data: [...hundred megs of lines]} is annoying for various parsers.

I'm quite a fan of parquet, but never expect to receive that from a client (alas).

0 comments

Izkata1y ago

> JSON that's in the form {data: [...hundred megs of lines]} is annoying for various parsers.

One reason this became common was a simple protection against json hijacking: https://haacked.com/archive/2009/06/25/json-hijacking.aspx/

cogman101y ago

Parquet should get the praise. It's simply awesome.

It's what I'd pick for tabular data exchange.

A recent problem I solved with it and duckdb allowed me to query and share a 3M record dataset. The size? 50M. And my queries all ran subsecond. You just aren't going to get that sort of compression and query-ability with a csv.

j / k navigate · click thread line to collapse