This is a mental operation, not physically possible. The difference is real: for example, charge inside a perfect conductor cavity generates electrostatic field outside. A charge outside perfect conductor cavity does not generate any static field inside.