That is fine when done at the individual level. What people are complaining about is when the platform makes that decision for you. It is fine to block a phone number from calling you, it is not fine to block people from being able to use a phone (by means of blocking access to all phone networks) without a court decision.

Now individual networks are private companies and might chose to have stricter moderation, but 1) that should place them in a different legal category, and 2) there should be enough competition to provide valid alternatives to the people being blocked from such networks, otherwise it risks abuse of dominant position, given the monopoly of access.