In order to move objects you need to stop the world and update everything that will ever point to them: pointers, pointer aliases, arithmetic bases, arithmetic offsets. This quickly becomes intractable. It's not strictly speaking just pointers themselves, but the fact that pointers can be used arithmetically in various ways, even though the most obvious ways are disallowed. The obvious example is unsafe.Pointer and uintptr, but that has some guards, for example you'll get an error converting from a uintptr variable to an unsafe.Pointer, but mix in some slices, or reflect usage and you can quickly get into shaky territory. I believe you can achieve badness even without using the unsafe package.