Deep in the weeds here, but the plural of a single species of fish is 'fish', while multiple species of fish are 'fishes'.
'Swimming with the fishes' means you must have wound up on the wrong side of a boat in a diverse body of water. It would be a shame to have to 'swim with the fish'. So ambiguous.