undefined | Better HN

0 pointsknome7y ago0 comments

> A 0 byte is perfectly acceptable in a utf8 string (or any unicode string, really)

What? My understanding was that utf8 was crafted specifically so that the only null byte in it was literally NUL. That all normal human language described by a utf8 string will never contain a NUL. They're comparable to C strings in that way, where it can be used safely as an end of string marker. If you have embedded NULs, it's not really utf8, is it?

0 comments

masklinn7y ago

> They're comparable to C strings in that way, where it can be used safely as an end of string marker. If you have embedded NULs, it's not really utf8, is it?

It is. NUL is a C-string convention, as far as unicode is concerned NULL (U+0000) is a perfectly normal codepoint (very much unlike e.g. the U+D800–U+DFFF range).

Dylan168077y ago

> My understanding was that utf8 was crafted specifically so that the only null byte in it was literally NUL.

Correct.

> That all normal human language described by a utf8 string will never contain a NUL.

Correct.

> If you have embedded NULs, it's not really utf8, is it?

Incorrect.

NUL is a valid character. If you accept arbitrary utf-8, or arbitrary ascii, or arbitrary 8859-1, then there might be embedded NUL. You can filter them out if you want, but they're not invalid.

paavoova7y ago

It's invalid for unix filenames to have a null character. Therefore, if your application is printing filenames in their unicode representation, it doesn't ever need to consider there to be a null byte. This of course isn't an arbitrary case, but it shows one can make assumptions regardless of the "validity" of a character. I believe for most cases of arbitrary input, the correct and safe thing to do is to assume a byte stream of unknown encoding.

Thiez7y ago

Since we arrived on this null-character discussion by considering text manipulation in C, I suspect most comments in this thread are made in the assumption that the text must be manipulated in some way (mine are!), so treating it as a byte stream of unknown encoding doesn't really solve the problem.

While null in filenames may be forbidden on Unix (and also on Windows), there are more exotic systems where it is allowed [1]. When writing portable software it's probably best not to make assumptions about what characters will never be in a filename.

Naturally if you have a problem where you can get away with just moving bytes around and never making assumptions about its contents then that is a great solution.

[1]: https://en.wikipedia.org/wiki/Filename#Comparison_of_filenam...

paulddraper7y ago

It's also invalid for filenames to have a slash, but I don't think that's very relevant to the discussion at hand.

j / k navigate · click thread line to collapse

0 comments

masklinn7y ago

> They're comparable to C strings in that way, where it can be used safely as an end of string marker. If you have embedded NULs, it's not really utf8, is it?

It is. NUL is a C-string convention, as far as unicode is concerned NULL (U+0000) is a perfectly normal codepoint (very much unlike e.g. the U+D800–U+DFFF range).

Dylan168077y ago

> My understanding was that utf8 was crafted specifically so that the only null byte in it was literally NUL.

Correct.

> That all normal human language described by a utf8 string will never contain a NUL.

Correct.

> If you have embedded NULs, it's not really utf8, is it?

Incorrect.

NUL is a valid character. If you accept arbitrary utf-8, or arbitrary ascii, or arbitrary 8859-1, then there might be embedded NUL. You can filter them out if you want, but they're not invalid.

paavoova7y ago

Thiez7y ago

Naturally if you have a problem where you can get away with just moving bytes around and never making assumptions about its contents then that is a great solution.

[1]: https://en.wikipedia.org/wiki/Filename#Comparison_of_filenam...

paulddraper7y ago

It's also invalid for filenames to have a slash, but I don't think that's very relevant to the discussion at hand.

j / k navigate · click thread line to collapse