I think there's a difference between doing this in your office, and an office you share with others.

If I needed to stay late to study, or grade assignments, or whatnot, I would be pretty annoyed if my office-mate asked me to leave because it was his bed-time.