Would you say that, or would you play along? Would you hold your "deep annoyance" as a permanent grudge? If you'd play along and not hold the grudge indefinitely then from their perspective you'd appear just like the (in my opinion more reasonable) friends who appreciated it, and so it's probably better to assume that situation.
but if I won't hear from him again for years and all of a sudden he texts me without any care then yea I'd hold that grudge and try to get him out of my life