I believe you might be limited by your imagination just how bad out of tree third party drivers can be. We're talking about hardware with DMA access here. It's trivial to create a situation that can not be fixed in the kernel (without breaking other things).
If you are running some generic PC hardware to write this comment, chances are that at least one tiny part somewhere which is dependent on some specific obscure timing to come up properly which just happens to work because someone inserted a small delay somewhere.
