Normally, the brain adapts to the warping, and it also means (as I didn't notice before the doc told me) that I have a tendency to turn my head rather than turn my eyes.
But with the "warp" constantly changing, the brain cannot adapt to it and it calculates the wrong trajectory for a flying object.
Hence my utter failure at hitting baseballs and tennis balls.
The doc opened by saying "you must be bad at baseball!", guessing correctly, and then explained all this to me. He also said I must suffer from severe headaches because of it, and accused me of lying about it when I denied it(!).
Contact lenses solve the problem by moving with your eye, but I tried them and didn't like the nuisance of contacts. (This is not an issue if you have only near or far sightedness.) I could get Lasik eye surgery, but I don't even want to take a small risk of damaging my eyes.