You'd instantly have to deal with people losing their keys, people damaging their keys, people's pets eating, digesting and defecating the keys, fire/floodwater/storms/earthquakes/other natural or man-made disasters destroying the keys, keys getting damaged by ESD or cosmic radiation, people stealing other people's keys for extortion or abuse... spread any technology over millions of people and you will experience all sorts of failure modes that you haven't even thought of.

All of these failure modes need some sort of "customer support" to work out, otherwise they'll not be used by users at all or they'll lead to shitstorms when people are locked out of their identity. And if the customer support makes errors or gets bribed, you'll get shitstormed too.

And allowing people to back-up their keys isn't an option either because that defeats the purpose of why you have an HSM anyway.

Security is hard, PKI is even harder.