There's a couple of different scoring rules that would be interesting, I think.
Another one might be: What's the ratio between the area of the country and the smallest rectangle which bounds the country. This would "punish" countries with protuberances more than ones with "in-cuts", which "feels right" to me.
For some reason, your use of "'punish' countries" made me imagine a world where improving this score gets taken seriously by the peoples of the world, leading to countries gradually becoming more rectangular over time, and many terrible wars along the way.
Alternatively, an era of peace and cultural exchange as countries agree to swap territories to become more rectangular, leading to an influx of new immigrants to both countries and greater understanding that this cultural mixing brings.
That's great, never heard of that before. When I search for that term, the first hit is a Reddit thread where the first commenter expresses his desire to "drop-kick" people who do it. Strong emotions! Maybe we can create this world after all.
I wonder how many actual wars in the past have been started because someone didn't like how the border looked on a map. It has to be non-zero.