Practically speaking I rarely find GC pauses to be an issue, neither latency wise nor speed wise. Though of course that could be due to
1. I don't need low latency in research work,
2. I rarely work with massive complex data structures filling all my RAM where the GC has to scan the whole heap every time it runs, and
3. GC may have indirect performance effects that are not measures as part of GC runs, e.g. by fragmenting active memory more.
