It's also the reflect on the value part which slows down things. It could be improved but it's hard to do that without breaking interface. It's so widely used that a major bump wouldn't that useful.
I'd expect it to be mostly called on map literals, at least a compile time constant set of keys. That should be amenable to a targeted compiler optimisation.