I have cantaloupe/honeydew but no problem with avocado or latex so I don't suspect a connection here. Raw bell peppers sometimes bother me.
Allergies are often funny, at least for me. Exposed to a single allergen usually results in no reaction. But many, many small exposures to diverse allergens adds up and can trigger a disproportionate response.
I have a friend who's food-related allergies (non-anaphylactic) disappeared after removing carpet flooring in their home.