I've done both and I'll add that the one thing I miss about Yocto is that it could package up an SDK with installer that could be deployed on a different machine. With a single install you have the correct crosstools, libraries, and headers to build directly for target. And when we used to develop with Qt that was a huge advantage in helping others get started.

But now I use Buildroot and I get things done without all the extra anxiety.