There are many schools around the world with much harder and more technology intensive curriculum than MIT. If you disagree with that statement, then you do see "value in competing over whose classes were the hardest." If you do not disagree with that statement, then you must agree that the "MIT experience" you describe can be gotten in many schools other than MIT. F.e, I also worked 10+ h/day 7 days/week year round at university and also felt that getting A grades were impossible.
I think the professor is making a great point by comparing math to sports. Yes, you can go to a sports university but you don't have to. The way to get good at it is by busting your ass of (and having good genetics). It's your own effort that counts, not the name of the institution, imho.