My doctor told me I should stop smoking.
There are a lot of ways to die outside of lung cancer though. Going down this path, it would be safest to not drink alcohol, not drive a car, or really just not live life at all.
My point here is that there are some things that have outsized impacts and can be avoided in isolation. Smoking is like that for health.
Javascript, ActiveX, java web applets, flash, any other way of executing arbitrary turing-complete remote code on my local machine directly, those are all vastly more likely to lead to CVEs than HTML parsers, image parsers, and other functionalities of browsers.
It's perfectly possible to identity and eliminate larger attack surfaces without slippery-sloping yourself into not being able to take smaller risks.