We've been storing ssh keys directly on our yubikeys since before passkeys were a thing.
Not only is it clearly more secure it's also been a usability lift. Plugin your yubikey, start an ssh agent, and run ssh-add -K to get all your resident keys added to your current session.