Historically the backspace key is literally that, moving the typewriter's carriage back (when the space key moves it forward).
On current computers the space key now inserts a "space" character instead of simply moving the cursor a step right.
Backspace isn't symmetric to that anymore, as it doesn't even insert a space back but deletes backwards, so it is (pedantically) a misnomer.
And this is true for "modern" computers, older ones were closer to teletypes and would move back for overwrite (e.g try vi compat mode).