Walking onto a set without permission is inconsiderate and disruptive because the crew is working very, very hard to make something happen and his presence isn't helpful. He effectively got lucky that he didn't mess something up and accidentally force the crew to work longer and harder than they have to. There's a reason the entire job of some production assistants is just to keep random people off set.