The problem here is identical to the problem of pointers to array elements in C after a 'realloc', except that Go at least guarantees that you're not going to modify some other object's memory.
Of course, since append neither guarantees nor prevents a copy, the semantics of modifying a value through a pointer to a slice element after an append are unspecified, so it is not a useful construct.