I've learned how to drive school buses, and I absolutely agree with your opinion. My first day on the job was my last (well, I drove a bit the next day before refusing to drive in a situation I was not comfortable in); it was just way too stressful thinking about those kids and how any mistake I make could hurt them.

Edit: clarified my actual quit story.