Oh god please don't. I worked with a startup whose "lead" (i.e. the person with the most seniority, not the smartest) insisted the whole front-end could be done with plain JS ("angular, jquery, etc... just useless bloat!!!" said the lead developer). What resulted was the biggest rats-nest soup of untraceable mess I've seen in a while. The only person on the team who understood the system was the "lead" developer. Changes took forever. Bug count was through the roof.
When you say "you don't need a framework", what you are really saying is "I'll build my own framework". Because you will be building a framework. And trust me, your framework will not be as good as what is on the market. And when you leave, all developers who didn't leave the company will rip all your shit out and replace it with a "real" framework -- all while cursing your name under their breath.
Your job at a startup is to get product-market fit. Not invent your own framework.