I don't think it would work in most situations. Construction site? Sure. Sporting event? Probably.
But in most of the places a person might try to go, I personally would avoid a high-vis vest at all costs.
I wasn't management, I was just new on site and had come from a planning meeting, and I was pretty sure the next guy I needed to meet was in the belly of the plant in a location I only had the vaguest idea of, but I knew it was gonna be a long walk.
I ended up walking the length of the place twice (almost a mile) before I decided I'd had enough sightseeing (but what sightseeing it was!) and and actually asked someone, who made a wisecrack about how he helps so many visiting managers in this joint they should make him a manager himself! I decided against correcting him.
Over the coming weeks at that plant, I wore a T-shirt and older hi-viz if I wanted to blend in as a worker, or a button-down shirt and the crisp hi-viz if I wanted to wander. As long as I kept abreast of my assigned duties, nobody sweated the details, and it was better than any museum of science and industry I've ever paid to get into. Had a few more folks make gentle cracks about how I must be a manager of some other group over to see what this group does, I'd ask a few questions and be on my way.
Likewise for jobsites and urbex, I have a hard-hat covered in stickers, and one that's so pristine I keep it in a pillowcase. Perceived wear is an important component of The Look™.
some time ago i couchsurfed with the owner or manager of a steel processing plant. he gave me a tour. one of the most memorable experiences.