The second is that it's a lot of code and files to make when we only have one import to hack something quickly. For a full library it's probably a very cool feature though.
The third one is more related to my use case. I made a binding with a non-C codebase, and Swift cannot directly import C functions AFAIK, so I had to write C code in addition to the Swift one to make headers. It's not impossible, but it does take time to write all of that in a third programming language.
The fourth problem is related to library imports. In SPM, you either import a system library, or an xcframework. If your app targets any non-Apple platform, xcframeworks are not an option, so if you made a binding to your own lib that's annoying as it is probably not installed while you're building. And even if you're targeting Apple platforms, making an xcframework without Xcode is not well documented.
I probably missed a few problems as my experience is not very extensive, and there are probably better ways to achieve some of the things I wanted to do, but there isn't much documentation about all of that online anyway as far as I saw.