That's what I typically do.
I'll make sure that the libraries I rely on are either ridiculously well proven, or easily replaceable. If they are neither of those things I implement a new one myself from stuff that is, while no doubt just as buggy, it's at least code I can easily fix myself if it breaks.
Usually it's not that much work. I'll just implement what I actually need. Many open source projects are hilariously over-designed and have significant feature creep. If you cut that out you can replace the critical components with not a lot of code and not a lot of hassle.
If it can't be trivially re-implemented, I've been known to fork an existing project and take a grinder to it until it does what I want, usually axing away most of what I don't need to only expose the code I need.