I agree that's adequate, in the sense that keeping the an SSH key as a password-protected file on disk is adequate, and having it be a password-protected secret in the TPM is no less secure than that.

But the whole point of binding a key to hardware is to be secure even if a remote attacker has gotten root on your machine. An attacker with root can simply replace the software that reads your PIN with a modified version that also saves it somewhere. Then they can use the key whenever your computer is online, even if they can't copy the key off. And although that's a bit limiting, once they've SSHed to a host as me once they can add their own key to authorized_keys in many cases.

That's why Yubikeys and U2F keys and suchlike have a physical button.

TPMs would be a lot more useful if the spec had mandated a physical button for user presence.