I don't get your reasoning (mainly because all you've provided none). I wrote games, we used Boost heavily, on the PC, Xbox360, DS, Wii and PSP. I mean really heavily. Boost.Thread, Boost.SmartPtr, Boost.Bind, Boost.Function, Boost.MultiSet, Boost.Array, Boost.PreProcessor, Boost.LexicalCast... (more)...
We turned off exceptions (BOOST_NO_EXCEPTION) because the Wii and the DS didn't support exceptions. That was it. We never had a problem with spurious assertions, nor with unexpected behaviour from a lack of assertions. Compile times were about 5 to 7 minutes for partial builds. A full build was longer (I forget), something like 15. This is a huge code base, remember.
I honestly have no idea what you're talking about. It's certainly not a "general rule of thumb".