You are right. It reassess the order of windows when you change application. And here I just thought I was bad at keeping track of my windows.
Another thing I noticed just now is you can browse to no-window-selected using cmd+`
As far as I can tell, the logic is the following, with numbers representing order of creation, with 0 being no window:
[3,2,1,0]
If you switch app with window 3 selected, the order is maintained. If you switch with window 2 selected, the new order will be
[2,3,1,0]
if you switch with window 1 selected, the new order will be
[1,2,3,0]
If you switch with window 0 (no window) selected the new order will be
[0,1,2,3]