Hah :) I don't mean that malloc itself is fast, I mean that having non-jvm heap memory is fast.

A permanent memory block on the JVM heap can't use pointers to refer to it, since GC moves objects around. And even though those blocks will never be collected, they make up additional work for the GC to track.