This is not necessarily the case. Sometimes you lack confidence with good reason because you are (currently, potentially will always be) unsuited to the task at hand, so you become nervous and stressed. There is nothing wrong with this either. Sometimes, it is worthwhile to persevere other times it isn't.
The hard thing is to know when to persevere and when to quit.