The risk is that over time Cocoapods will no longer work to integrate dependencies with ways Xcode wants them to be.
Switching to SPM is a massive undertaking for a large project with many intertwined dependencies and configuration options. It's far less configurable than Cocoapods with basically no scripting options during install.
Their progress already is pretty decent!