Makes sense that this diet wouldn't work for you - but I think using this argument is sort of like arguing that peanuts are unhealthy because some people are allergic to them.
Fun Fact: You can let your potatoes cool down, and then re-heat them, to significantly lower the glycemic impact.