personally, I'm no savant, but as a child I really didn't have to work hard at anything in school. I was pushed up a grade, and was still at the top of that class, and bored, for years. i put in basically no effort in to any schooling for years, and was still, generally, way ahead of many other students who, looking back now, were struggling (this was in the day when students could be 'held back' to repeat a grade - I don't think that's done much today?).
i have learned plenty of skills, and some took years, and it's never ending. but some came - essentially - effortlessly (or appeared effortless relative to peers' efforts).
My claim is that the latter is a skill that nobody is born with. Autistic savants aren't some type of magic creature that know things just by virtue of being savants. They still have to go through the process of skill acquisition. Now that process may be accelerated compared to me or you, but I disagree with your claim of proficiency with zero effort, especially with skills that have shown to require thousands of hours of deliberate practice to establish proficiency.
I think the key is in your last statement: "or appeared effortless relative to peers' efforts". It seems you found yourself in an environment which didn't sufficiently challenge you. This would only argue that you should've been pushed up to more challenging AP/honors classes. This would again have the effect of placing you in a higher standing compared to your peers. So if both you and your peers would be pushed to your true potential, it seems consistent with your statements to say your performance/output would've been superior.
So why should colleges deny you entry because of your ability to be proficient in the system they've set up?