This isn't just a matter of things "seeming". It is quite literally an implication of Safe Rust works => field offsets must be aligned. There is no other way for safe Rust to be safe, other than alignment not mattering at all because all accesses are careful to pessimistically not rely on it.
I am sorry, but this hypothesis is just completely outside the Overton Window.