A map or pretty much any nontrivial data structure is not a flat region.
In any event, waiting to optimize until a profiler tells you to is a reasonable practice as long as you pay attention to scaling. It’s very easy to write, for example, a JSON parser that performs fine in small tests and has a nice small n^2 coefficient. And then someone throws in a bigger input than you tested and your game takes ten minutes to load.