Remember, they don't know which part of the framework you are using or not, or which parts are important or not, so even if you are using it for one little thing, they probably have to learn most of it to understand how it interacts. Since so many frameworks overload standard functionality, it is impossible to know how a framework interacts with your code until you know all of these things.
Using 20 frameworks is about as efficient use of your time as having one you built yourself. The sweet spot for a medium-sized application is using 2-4 (maybe a few more if they are really standard or do really specialized functionality - i.e., a PDF library) for the most critical components, and still maintaining the remaining quirks yourself.
So, for example, Ruby-on-Rails + Bootstrap + JQuery goes a very long way, and doesn't put an undue burden on people trying to get to know your code.
If your application become sufficiently complex that it can't be easily grokked by a decent dev, that's when you know it's time to switch.
Until then it's just over engineering (which the JS community admittedly loves).
I estimate that every 20% that a codebase grows it needs some amount of refactoring. Definitely anything that moves from skunkworks to production will need some pretty heavy rewrites, and will probably need a framework that better meets its newer needs. The question is how much cost was sunk in getting there. The refactor will have to happen no matter what.
Guy A believes his custom design is simple enough to comprehend by anyone. Guy B doesn't agree.
It doesn't really matter at the end of the day. If you work in a team environment, the toolset will be pre-determined already. You don't have a choice.