I would even go one step farther and say it's a mixture of both.
The classes at my uni could move at a more intense pace than other schools, and the problem sets and exams were also more difficult, given the academic aptitude of my class. Go on the MIT Open Courseware web site, and take a look at the physics problem sets to get an example. I'm not sure there are too many other freshman physics classes that require pulling all-nighters to solve the challenge problems.
Sure, I could always challenge myself more wherever environment I was in (I taught myself linear algebra and C++ coding in high school, even though nobody I knew in real-life could help), but there is a LOT said to be surrounded by peer pressure, mentorship from world-class professors, and other motivated, top-notch students with the same drive for success. I would not have pushed myself to do better to the extent that I did at my uni.