Not everybody feels the same way. It all depends on how your life progressed and your experiences as a child and teenager. Some people desire to one day be the father they never had... others had that father, so it is not an issue.
However, we all have insecurities... it may not be about love, but there is always something out there.