Calories have something to do with it because they are one part of the equation. Calories in does equal calories out. Calories In = Calories Stored + Calories Expended. But, the mistake is in thinking that the law of thermodynamics means that the type of calories in doesn't impact the ratio of calories stored/expended -- the variables are not independent. The content of the food you eat impacts: your hunger, your energy level, fat storage rates, and your metabolic rate, all of which mean that attempting to cut or add calories may not have the expected impact depending on what the food is.