It is true that hardware tokens without an integrated external I/O (not through the PC it is plugged into) are vulnerable to this type of attack during initial key setup. Maybe they should support using the indicator LED to morse code thumbprints or something, but if you can't trust I/O to the device during setup you're going to have a bad time. I will quote my original caveat here:
> Just plug it into a computer with a CPU you do trust first to get the official public key(s) for future verifications!
Your second point seems to be that the plaintext has to be exposed (either going in or coming out) at the USB/hardware interface level, which makes more sense to discuss. The sales page promises the following, but I didn't quickly find any details on how it works:
https://www.yubico.com/products/yubihsm/
Secure session between HSM and application
The integrity and privacy of commands and data in transit between the HSM and applications are protected using a mutually authenticated, integrity and confidentiality protected tunnel.