Most web servers don't really come with any built in defense against brute force attempts vs Basic Auth gates, so unless you've set something up to protect it, someone with enough time will eventually get in.
Only when the password is weak enough to bruteforce swiftly. It will take literally thousands of years to bruteforce strong passwords.
That's only correct if the password is weak. With enough entropy, it's practically impossible to brute force.
> just go to any old computer and ssh into my server
You've typed your password into a computer you don't control. Now it's gone. Same for plugging in the USB stick. The Yubikey approach mitigates that.
Assuming you want to do this, the best practice you can achieve is just making the password long.
You can mitigate it by using an MFA method that requires confirming on a separate device like a phone, but that's down to one layer of defense.
I use an SSH app on my phone for remote access, and I go over a VPN. SSH is not exposed to the public internet.
