I'd be willing to bet money that at least some of the formats in question aren't UTF-8, they are likely ASCII encoded against a character set or code page.
Then you have to read that codepage, and convert the necessary characters to their Unicode equivalents, and from there do you downcode to utf-8?
Does the language this library is written in support that translation? Are there modules to do that? Is the license for those module(s) necessary compatible?
Who's going to go through the different document versions to confirm, and adjust for the various encodings for non-ascii characters?
It's not as simple as saying "don't choke on unicode".
Length-prefixed byte arrays encoded using various code pages. There are a small number that excel uses: https://github.com/SheetJS/js-codepage/blob/master/excel.csv (the columns are CP#, mapping, single/double-byte)
> Does the language this library is written in support that translation? Are there modules to do that? Is the license for those module(s) necessary compatible?
If we can put together an Apache2-licensed module in JS in an afternoon (https://github.com/SheetJS/js-codepage) it can be done in python.
> Who's going to go through the different document versions to confirm, and adjust for the various encodings for non-ascii characters?
Someone already did that: https://github.com/SheetJS/test_files/tree/master/biff5 has artifacts for every language type
I thought Python 2 was Unicode-unfriendly. So not as easy as JS.
It's written in Python, which comes with support for pretty much every major encoding¹ out of the box, so yes.
¹: https://hg.python.org/cpython/file/cb94764bf8be/Lib/encoding...
In ten years it might be better.
Unicode isn't even hard: Use UTF-8. Don't try to measure the length of a string unless you're rendering that string and measuring the length in screen units like pixels. If you do those two things, that's 90% of the effort of making Unicode-safe software.
I think both views are valid. Those who don't know how to write Unicode-safe software shouldn't feel shamed into learning Unicode before releasing open source work. Those who already know Unicode should feel happy that they're making other people's lives easier.
