Let me try to explain - I've had cancer at a relatively young age, so heard that a lot. I wasn't visibly sick, i just knew there's a chance I'm gonna die in the foreseeable future from it.
Unless the person saying it had access to some experimental treatment my doctors don't know about, there really was nothing they could have done to help me. I knew it and they knew it.
So while I didn't get angry at them (because I knew they said it because didn't know how to react differently), it did annoy me as it was basically an empty promise.