People say this a lot, and it makes me wonder if they've ever worked on a team working on the same codebase. And if they do, what secret sauce they're drinking to not go insane.
In my experience, when people on a team use libraries at their whim, the coding styles diverge wildly from feature to feature. One person decides they're going to write all their AJAX callbacks in an anonymous function; another likes to reference a function directly; another likes to both; and yet another uses promises.
And as soon as someone needs to fix someone else's project, they do some combination of (a) getting wildly frustrated with this other coding style, (b) rewire their mental model to adapt to this new style, (c) spend a week rewriting it in their own style, (d) introduce a ton of bugs in the process, or (e) by some miracle, get it to work perfectly.
So to combat this, the team works out coding standards and a style guide. You must tab with two spaces! You must pass a real Javascript function to your callbacks! You must separate your state from your view!
And of course, once your style guide is big enough, lo and behold: you have your own hand-rolled framework. And hopefully it's tight enough that everyone writes in the same style, but is still expressive, and is still simple enough for a new team member to learn quickly.
So if you're just starting a codebase, it makes complete sense to pick a framework on the outset because people have already gone through that headache for you.