I think you overestimate how much ordinary people care / know about their online privacy. I think almost nobody would leave. That said, I myself wouldn't be too happy about it, but I would try to find some other solution instead of leaving LinkedIn.

LinkedIn also has an option 'Select what others see when you've viewed their profile' in account privacy settings which one can set to 'Anonymous'.