It's not such an easy decision. Sure if you have more reps, each constituent arguably has more access. The flip side is that they have more access to a less powerful rep. If you had 10,000 people in congress, sure it'd be a lot easier to get in touch with one of them, but 1 person in a group of 10,000 doesn't have much influence, so how much is that interaction really worth?
Even at 435, unless your rep is a senior member with some important committee assignments and they happen to be in the majority party, getting the ear of your rep isn't worth all that much.