One key thing missing here is there's not enough nuance in the outputs to really control a vehicle. For example, "go forward" -- how fast should we be going forward? Does this mean floor the accelerator? Probably not, but that's not captured in the output here.
A couple other key scenarios to think about:
- A pedestrian jumps in front of the car. What happens?
- It's driving on a freeway and need to exit -- what actions are needed to make that happen?
