You can change your phone number though. What if you change it so frequently that it's no longer a key identifier? Why not have a system where you can receive phone calls without a phone number? I believe some of the solution here is to stop having globally unique identifiers.
Burner numbers are close, but if you use the same one for all your contacts it still becomes an identifier, even if you rotate it. An alternative is a proxy service where you get issued a unique pin for each person-to-person connection, so the same phone number gets used by multiple parties. Those both use the existing phone network in better ways. With a VoIP system you don't need numbers, you just don't want to trade them for usernames or some other global identifier. I implemented one approach with Severus, but I'm also looking for alternatives that increase privacy.