Like you said, high-tech outdoors is troublesome. If you want to get a head start on this sort of thing buy an outdoor robot chasis (or make one if you can hold a screwdriver, it'll save you a grand or two). Throw on a kinect, all the cameras you can fit and all the lidar you can afford.

First, teach it what rain is and how to quickly get back to the safety of the charging port. Second, make it navigate around the garden without destroying the plants. Finally, teach it what to look for (pests, damage, ripeness). Don't bother giving it manipulators, just have it log findings and generate reports. When you get home, it can tell you exactly which plants need attention.