This is solved by statefulness: the router/firewall can be told to drop by default any unsolicited connections.
It's how things work with IPv6, which doesn't have NAT (by default): just because a host has a globally routable address does not mean it is reachable by default.