In short, it's hard not to feel like you just got baited and switched, even if the withdrawal is for a good reason. "But you'll still be at the ceremony!" isn't a good argument because that wasn't the original invitation, and because you're putting them in a very uncomfortable position having to decide whether to continue to go or not.
For context, I've spoken at over 100 conference events. There are very significant differences in terms of the keynote speakers and regular speakers — generally in compensation, status, time, and prominence, at least for conferences that are larger than community events. Not all of these may not be obvious to attendees, but overall it's a fairly serious snub to pull the rug out from under someone like this.