In weight lifting you have this concept of "hard gainers," who are generally men who want to gain muscle mass / weight, but can't seem to, "no matter how much they eat." A common refrain from them is something like "I ate a whole pizza on my own, and still nothing!"

Well, the moment they start actually counting their calories, it's the same story you just told. Sure, they ate a whole pizza on Saturday, but also nothing much else that day, and they ended up short of their caloric needs, and then every other day that week was even less, minus the occasional splurge that also didn't bring their average up high enough.

This goes for people who "can't" lose weight too -- when they start counting calories they are surprised to find that they eat a shitload of calories they didn't know about, most of which are empty of nutrition.

One way to reframe the question of calories is to start considering calories per week, instead of per day. You start to get the picture either way--whether you're trying to gain or lose--when you're thousands off your goal by day 3 and realize what a big adjustment you'll have to make during the next 4 days to make your count right.