Are there non-ascii compatible encodings that are default in any OSes? With ascii-incompatible system/terminal encoding a lot of software will stop working. Strange things happen, but this looks like a theoretical issue, and ascii looks safe. In Python 2.x __str__ of all standard container types are ASCII-only (even if elements has non-ascii __str__), and __repr__ of standard objects is also ASCII-only as far as I can tell. ASCII-only is an option, and it is not uncommon and relatively safe (but it has its own issues of course).

It was exactly this unicode_literals property (turning everything into unicode) that helped to reveal bugs :) For example, models were trained on bytestrings under Python 2.x, and nobody remembers what was the encoding of the text models were trained on. This was unnoticed for several years because instead of raising an exception functions just handled some egde cases (e.g. unicode punctuation) in a suboptimal way. This leads to almost correct results, but with less accuracy/precision/recall. After changing to "unicode everywhere" the issue became visible.

The issue was not with cancer-like turning text into unicode, issue was with the code that works with text and doesn't support unicode. Python 2.x standard library has such APIs, and this causes troubles, but I don't see how it is a bug in the code that works with text and returns unicode.

What I'm writing are common words and a standard "unicode mantra", but anyways.

We could say "programmers should just handle encodings properly, and unicode_literals have nothing to do with this", but this doesn't always work. "Unicode everywhere" makes some code changes necessary, but some of these changes reveal real bugs.

Another story: I took 2 different courses from 2 different top-notch universities at coursera.org where instructors gave us starter code (written in Python 2.x) for programming assignments. The code was not bad, but there were many cases of incorrect encodings handling in most of the provided files (such errors that would be impossible in Python 3.x) - this was the code that was supposed to teach students something (including Python programming).

What I like about unicode_literals is that it makes things more consistent and easier (at least for me) to reason about: if variable is unicode under 3.x, it is unicode in 2.x, the same applies to bytestrings. In cases where different behaviour is necessary (e.g. because of non-unicode API in 2.x stdlib), explicit str("foo") is used; otherwise code is written in Python 3.x and works with the same semantics under Python 2.x.

Just curious, what newcomer issues are you talking about, and who do you mean by "newcomers"?