Fully powering off the camera will help, but I imagine with direct sunlight going through the lens and hitting the sensor the heat could still damage it, despite it not recording.
I have a sky camera (use it for weather and sky observations; my airspace is a bit busy for easy meteor observations). So far my sensor seems to be fine despite an unblinking gaze upwards.
