It sounds like you, too, were doing application development rather than library development. By which I mean that -- even if you were developing a "library" -- you more or less knew where & how that library was going be used in the overall system/application.
That's all fine and dandy for your case, but not all software development has the luxury of being so limited in scope. Testing the application fundamentally misses a lot more edge cases than a unit test would ever miss. And setup/teardown takes so much longer when every single change in some part of the codebase requires you to re-test the entire application.
When your project gets bigger or the scope becomes open-ended (think: you're writing a library for arbitrary users, like Boost.Regex), you literally have no application or higher-level code to test the "integration" against -- unit tests are your only option. How else are you going to test something like regex_match?
> what the @#$@Q#$ is a unit anyway?
https://res.cloudinary.com/practicaldev/image/fetch/s--S_Bl5...
P.S. I have to also wonder, how much bigger was the entire engineering team compared to the 3-7 people you mention? And if it was significantly bigger, how often were they allowed to make changes to your team's code? It seems to me you probably tight control over your code and it didn't see much flux from other engineers. Which, again, is quite a luxury and not scalable.