Where it breaks down is for group conversations. If Person A won't use Signal and Person B won't use WhatsApp, you can't easily have group communications. And it only gets worse as the number of people in the group goes up.

FWIW, as far as I ever could tell, Facebook did this correctly: the only real thing is letting a business have an account without a phone number; they then provide the software you can run on your server to be a WhatsApp client, so all of your user's messages are then end-to-end encrypted to your business. Yes: later on they decided they'd get in the business of offering a "hosted client"--which meant that, technically, if you used that service, they could see the messages, which caused a change to their terms of service, as a blanket statement that Facebook can't ever see messages isn't technically true anymore, which Signal threw a ton of FUD at :/--but anyone could have offered that service before (and could right now also for Signal).