> just go to any old computer and ssh into my server
You've typed your password into a computer you don't control. Now it's gone. Same for plugging in the USB stick. The Yubikey approach mitigates that.
Assuming you want to do this, the best practice you can achieve is just making the password long.