You should be right! But you are misidentifying the problem.
The problem here is not the norm, it's the exceptions. Some drivers don't slow down or pay attention. Those are the ones that cause all the risk. While normal drivers behave better at intersections, a pedestrian can't trust that every driver will do that.
In a mid block crossing, you don't have to rely on drivers behaving a certain way, so the predictability is increased and the overall result is safer.