But this is the crux. What sorts of aliasing are permitted in unsafe Rust? I think we kind of don't know; that's what the "stacked borrow" effort is trying to answer.
We know pretty well for the vast majority of patterns whether it violates the aliasing rules or not. There may be some corner cases that are not yet decided on, but if you just consider those as UB for now, you are perfectly fine.