> Text editing and rendering

Unless you're working entirely in fixed point characters (and you probably aren't, given that even fixed-width systems like terminal emulators use double-wide glyphs sometimes), you need to know the value of each character to know its width. That involves the same linear scan over the string that is required to calculate the number of glyphs in a variable-width encoding.

If you implement naïve Aho-Corasick text search over one-byte characters, it works without modification on UTF-8 text. It does not ignore combining characters, but UCS-2 also features combining characters (c.f. other comments in this same thread), so no matter what encoding you use, you must first normalize the Unicode text and the search string before you compare for equivalence (or compatibility, which is a looser notion than equality for Unicode code point sequences.)

> Text editing and rendering. Some parts of the system cannot simply treat Unicode text as an opaque binary hunk of information.

Except these parts of the system have to work on unicode glyphs (rendered characters) which will span multiple codepoints anyway, so counting codepoints remains pointless. The only use it has is knowing how much codepoints you have. Yay.

How does fast character counting help with full-text search?

Could you not have communicated your examples without the hostility?

I think you misunderstand. I wasn't asking why Unicode characters should be counted instead of bytes or ASCII characters, I was asking why you would even want to count characters at all.

> I fucking hate you ascii-centric ignorant morons

Nice.

> You ignorant, arrogant fuck.

This is why I quit posting under an alias, so I wouldn't be tempted to say such things.

> display welcome message character by character fro left to right

UTF-16/UCS-4/UCS-2 doesn't solve anything here. Counting characters doesn't help. For example, imagine if you try to print Korean character-by-character. You might get some garbage like this:

    ᄋ
    아
    안
    안ᄂ
    안녀
    안녕
    안녕ᄒ
    안녕하
    안녕하ᄉ
    안녕하세
    안녕하세ᄋ
    안녕하세요

Fixed width encodings do not solve this problem, and UTF-8 does not make this problem more difficult. I am honestly curious why you would need to count characters -- at all -- except for posting to Twitter.

Splitting on characters is garbage. (This example was done in Python 3, so everything is properly encoded, and there is no need to use the 'u' prefix. The 'u' prefix is a nop in Python 3. It is only there for Python 2.x compatibility.)

    >>> x
    '안녕하세요'
    >>> x[2:4]
    'ᆫᄂ'

I tried in the Google Chrome console, too:

    > '안녕하세요'.substr(2,2)
    "하세"
    > '안녕하세요'.substr(2,2)
    "ᆫᄂ"

I'm not even leaving the BMP and it's broken! You seem to be blaming encoding issues but I don't have any issues with encoding. It doesn't matter if Chrome uses UCS-2 or Python uses UCS-4 or UCS-2, what's happening here is entirely expected, and it has everything to do with Jamo and nothing to do with encodings.

    >>> a = '안녕하세요'
    >>> b = '안녕하세요'
    # They only look the same
    >>> len(a)
    5
    >>> len(b)
    12
    >>> def p(x):
    ...     return ' '.join(
                'U+{:04X}'.format(ord(c)) for c in x)
    
    >>> print(' '.join('U+{:04X}'.format(ord(c))
              for c in b))
    >>> print(p(a))
    U+C548 U+B155 U+D558 U+C138 U+C694
    >>> print(p(b))
    U+110B U+1161 U+11AB U+1102 U+1167 U+11BC U+1112 U+1161 U+1109 U+1166 U+110B U+116D

See? Expected, broken behavior you get when splitting on character boundaries.

If you think you can split on character boundaries, you are living in an ASCII world. Unicode does not work that way. Don't think that normalization will solve anything either. (Okay, normalization solves some problems. But it is not a panacea. Some languages have grapheme clusters that cannot be precomposed.)

Fixed-width may be faster for splitting on character boundaries, but splitting on character boundaries only works in the ASCII world.