It should give you pause if you find that you are spending much of your time fighting the GC.
Perhaps you're doing something very wrong in terms of object allocation and lifetime. Perhaps this GC/memory performance debugging time is well spent and expected - just as you might optimize memory usage (and VM and cache usage) to squeeze out better performance from a C++ program. Or perhaps .NET is not well suited to your requirements.
