>The rust code was running circles around their Swift ones even when we tried to keep implementations the same.
I've done Advent of Code a few years -- even Javascript implementations, if using a good (optimal) algorithm, are highly performant, so I'm suspicious of the claim. In most AoC problems, if your code is observably different between languages, it's the fault of the algorithm, regardless of language. But perhaps you are referring to well-profiled differences, even if there are no observable differences.
That said, in projects other than AoC I've compared Swift to C++ and it's hard to deny that a low-level language is faster than Swift, but Swift itself is certainly fast compared to most dynamically typed or interpreted languages like Python, JS, Ruby, etc. which are vastly slower than anything in Swift.