Yes, but at the cost of portability. You won't be able to run the same model on Android for instance.
You're still debugging in two places; in Swift and Python. Debugging Swift is probably easier than C++ though.
I think using Swift is a valid solution for special cases, but not the best solution for most cases. The TF authors already provide a suitable, general solution in the form of custom TF ops.
And if you don't need a custom kernel, and the chances are you don't, then stick with pure Python for maximum ease of use and portability.