Two reasons come to mind:

1) Since I was a kid, sea-side vacations in the summer were common-place to the point that I actually learned how to swim in the sea. I still remember how surprised I was at the difficulty of swimming in a pool where you are much less buoyant. No salt in the water and all that.

And 2) Maybe I just haven't swam in choppy enough water. But I do remember that when waves are a certain height above your head, having a surfboard at hand suddenly makes life so much easier. Although even then I didn't feel a particular propensity to inhaling water.

edit:

3) I learned several techniques for swimming in place and resting when you're tired. If push comes to shove, you can also just lay on the water pretty much indefinitely. This one doesn't work if the water is too choppy though.

4) I probably haven't swam in a super cold sea yet. Think the coldest I've done without a wetsuit was Monterey Bay last summer where I managed to skinny dip for some half an hour or so.