No, it wouldn't. Thunderbolt versions 3 and 4 are giving you 4 lanes of PCIe 3.0 on an external connector. You can easily get 3GB/s from an NVMe SSD in a Thunderbolt enclosure, which is the same performance as putting a PCIe 3.0 SSD into your computer.

We don't yet know how fast the SSDs are in the new Mac mini, but the M1 Mac mini was using ~3GB/s SSDs, which is the same speed you get from PCIe 3.0, and realistically.. that's plenty fast for this discussion regardless.