Not just letters! "^X" indicates "X, but with the control-bit flipped", where the control-bit is the left-most of ASCII's 7 bits.

    1001000 ASCII 'H'
    1000000 "ctrl bit"
    0001000 ASCII backspace

But like I said, it's not just letters, if you use `cat -v` to show all non-printable characters in this notation, you'll also see things like "^@" for the 0x00 null byte and "^[" for the ESC. All of the terminal color-code control sequences start with ESC, so if you pipe colorized output to `cat -v`, you'll see a lot of "^[".

I don't think that's the answer to the backspace=^h question, but HJKL is precisely an artifact of the keyboard that vi was first used on: https://en.wikipedia.org/wiki/File:KB_Terminal_ADM3A.svg / https://en.wikipedia.org/wiki/Vi#Creation - in addition to HJKL being labeled as the arrow keys on that keyboard, also note the far superior placement of escape and control where modern keyboards typically place tab and caps lock; this also made vi a lot more sensible.

(And of course that last bit is a whole topic unto itself; to this day, caps lock is the dumbest key on the whole keyboard to me - mostly useless, takes up prime real estate, and 100% worth remapping on every machine I control. But that's a rant for another day:])