Now this is scary.
I'm going to reveal some of my opsec but my password manager (pass(1)) does have yubikeys registered but it also accepts my GPG key. So even if I lose my yubikeys I can still unlock all the passwords, otp codes and everything I have in there.
I just can't feel comfortable with any other solution than my head being the final master key.