If you've seen some of the Lunar or Martian landing videos, you'll notice that it's very hard to tell the scale. Especially on the Moon, the lack of atmosphere to disturb the surface makes it fractal-like, which probably really messes with the CV algorithm. It'd work fine when high up, but as you approach for landing, it would probably struggle, especially for, say, estimating how far away the surface is.
