C# is better than many alternatives, but Swift is simply a better language in every regard.
C#'s GC is a constant issue for games, and it makes using things like LINQ nearly impossible since it does so much allocation. Swift is designed around automatic reference counting instead. C#'s structs work weirdly and are hard to use.
Swift is so much more ergonomic. C# has added a few nice things lately, but it's much more verbose than Swift.
Swift is also more focused on performance, and is always AOT compiled.
Unity has had to come up with some insane things to get C# to work across platforms – they have their own .NET IL to C++ compiler (https://docs.unity3d.com/Manual/IL2CPP.html) that doesn't always work right. They also have a SECOND custom C# compiler for high-performance programming (https://docs.unity3d.com/Packages/com.unity.burst@0.2/manual...).
I'm also very excited about Swift 6's upcoming opt-in Rust-like lower-level memory management which should help a ton for performance-critical code. (https://github.com/apple/swift/blob/main/docs/OwnershipManif...)