Facebook, Google, and the DOD all rely on yubikeys now to mitigate a lot of remote attack vectors. It is not perfect, but it is well audited by some of the best in the industry and as good as it gets atm. There has never been a situation where a yubikey (or other GPG smartcard) has allowed read access to private keys. The secure elements do their job.

As for this vuln, it was a flaw in pin validation, the secrets were never exposed. An attacker that has physical access to the system using the yubikey could simply wait and intercept the pin without this flaw. Any active agent no matter how it was initiated is subject to hijacking like this.

The pin protects against use when physically stolen, not when in active use by the owner. You get assurances your secrets are never copied, but no assurances they are not being used right in front of you by a third party on a machine you think is trustworthy.

The only way I know to mitigate an attacker hijacking keys like this without significantly inhibiting workflow is configuring automatic ejection and re-enumerate the device on touch. Then ssh talks to gpg agent which blocks and waits on key insertion. You touch it and it completes the ssh handshakes then it auto ejects the moment the operation is over.