If you believe it makes you unhappy, and not simply causing you stress, then you shouldn't go back. I realise your parents and friends might disapprove at first, but you should change their minds by showing them you can do what makes you happy and be successful.

It shouldn't make any difference in you social life, because if someone cares that much, they are probably snobby as fuck.

However, it might make a difference to you in later life. Russell Tovey (British actor) said he always feels ashamed for not having a degree in acting (or whatever it's called, I honestly forgot the term... Drama?) when people talked about which schools they went to. Your parents are probably just scared that you might develop self esteem issues like these.

I am not sure if it will limit your career choices, but you can always compensate with a kickass portfolio!