It's not clear to me that it's impossible. The ABI isn't a huge issue; templates are, though. I can see a potential path forward by using clang heavily, but it requires a lot of thought.
Isn't Swift looking heavily into C++ interop? It might make sense to collaborate with them to get it to work in both languages.