> The only real defense is to never associate your phone number with personal accounts which even then is often not possible.
Yes that's exactly right. If I don't trust a website to not use my phone number as the sole factor for recovery, then I should not use SMS 2FA on that site and I should not add my phone number to any part of my profile. If I know (how?) that the website won't use SMS for recovery, then SMS 2FA is better than nothing.
As a website owner, if I offer SMS 2FA auth and use SMS in isolation for recovery, then I'd want to stop using SMS for recovery. After that, removing SMS 2FA and not offering any second factor would weaken my security. I.E. SMS 2FA is weak but better than nothing. SMS single factor recovery is terrible, fix that ASAP.