I've wondered that too. What if in human pre-history (10,000+ years ago) modern rhinoviruses and coronaviruses that cause common colds, were once terribly deadly. We could just be the descendants of survivors who evolved immune systems that were able to deal with these viruses?
One of the techniques of trying to make vaccines is to attenuate it by exposing to to different hosts; so that it will mutate into a form that's less harmful to humans; kinda the opposite of what probably happened here, where a virus that was not deadly to either bats or pangolins but caused havoc when it got to us.