I used to believe issues like the Earth moving would kill you. But this model of time travel envisions space as a clean 3 dimensional box in which newtonian orbs move around in. Time travel is just moving the clock, so you need to make sure to account for changes in your personal orb between times. Wormholes already have all sorts of issues reconciling different initial frames on each end, and time travel suffers from those same thought experiments.
Instead of orbs in a box, I purpose a different mental model. Modern science indicates that our 3d dimensional world may be an illusion, and space is definitely not a uniform box in which orbs move. It's likely expanding, gravity definitely changes it, and it may all be a holographic projection. In these scenarios the distance between objects and their attraction may be an result of an underlying super structure. If time travel does exist, it would likely involve these pieces. Extending this, when traveling through time you may be able to 'surf' along the waves that large objects leave in space and time. That way time travel goes from "move the clock from X to Y" and more "travel in a way outside of our normal understanding by still confined by forces." This means you could stick to your orb without too much issue when traveling through time.
All I'm saying is that time travel likely won't be taking an absolute point in space time (x, y, z, t), and only making a single, discontinuous change to t.
