You also won't have access to the TOTP backup of emergency codes.. If you get hit on the head or have the trauma of your life you might not have a password.

These are tradeoffs that people use to balance security. Many people will lose to a sim swap despite not needing access to their life savings while on vacation with absolutely no proof of identity and a fallible memory.