>If they're complaining about your hours, then there's something else bothering them, but they're using this as a proxy[2].
This can also be a sign that the manager doesn't really know what you're doing and/or has not set clear goals for you, so they have nothing else to measure except your hours.
As VP Eng with an org of 35, my CEO was always giving me grief about the number of hours people on my team were in the office. I always pushed back with our milestone tracking. We are on target, and that's all that matters to me. People know what they need to get done, and that's what I want to manage, not hours in the office, vacation balances, or anything else. I'm not their parent. I set expectations, support them in meeting those expectations, and let them make the adult decision about their time.