If your connections crap and there’s packet loss, some of the sequence may be lost.
Avoiding replay attacks is another whole problem - you want the sequence to change based on a shared secret and time or something similar (eg: TOTP to agree the sequence).
Then you have to consider things like NAT…