Can you say more about the challenges of people moving behind you? I felt similarly but it had more to do with whether or not people would correctly interpret what I was doing / not doing. So for example I'll read something from a different subject to pull my brain out of an endless loop when I am not making progress on a problem. Is that goofing off? Sure it might look that way if you didn't ask but if you did ask you would get the full story. So can people own up to asking? Or do they run with their assumptions?
As a manager I like to know that folks are making progress against the things they are responsible for. Sometimes they have milestones, sometimes not, so I spend some time trying to understand that progress. If there is little progress and a lot of web surfing, that is a useful conversation to have (trying to be more productive). But if someone is spending their days checking in excellent code and leaves TMZ up on their monitor I'm totally fine with that too. It reminds me of an anecdote.
So my daughters had this tendency to do their homework in front of the television. At first glance it looked like more "TV watching" than "studying" and the conversation we had was about results vs consequences. I didn't care one way or the other if they spent time watching television, as long as the priority was to get their work done. The consequences of not getting that work done occurred whether I approved or didn't approve. But I was also quite clear that it was their choice and so there aren't any excuses for poor work if it is done while watching TV, its just the indicator that you can't really do both and expect to do good work. Conversely if you do both and your work is fine, then that is totally fine.
If I can develop a level of trust with someone that they realize what I care about is that they get done what they say they will (and that getting done both a reasonable amount of work and at good quality levels) then they also understand I have no issue with them apparently "goofing off."