The BBC would face significant cuts if that was the case. It would likely end up at a point where the quantity and quality of content was reduced due to less revenue, which then would cause additional people to cancel subscriptions.
This would be quite damaging for the UK as the BBC is a great way of projecting soft power, plus it provides quite a lot of great resources (as has been pointed out in other comments).
I'd be more in favour of a system like Germany has to be honest.
So I should pay the license fee to keep the quality of content I don't watch high? If the BBC's value proposition is so good, it will have no issue convincing the public of that.
You could hold a similar debate on every service the government currently provides, and you would essentially end up with toll roads everywhere and parents having to pay exorbitant amounts to educate their children properly.
The BBC is similar to these things in the sense that it provides good value for the taxpayer, and contributes quite significantly to society (even if it doesn't directly effect you in the sense of you consuming their content regularly).
