Construction workers have the second-highest number of workplace deaths (transportation workers are the highest) and among construction workers, falls are the most common cause of death.
Transport and construction are second and third in fatalities per worker-hour, with 'farming, fishing and forestry' the most dangerous per worker-hour.
So ladders aren't the most dangerous thing out there - but falls are pretty near the top of the list when it comes to workplace deaths.
That's not to say people can't use ladders safely - just if you're hauling a heavy ass drill up a ladder and it's stopping you using both hands, the cable's flapping around your legs, your pockets are full of sharp pointy tools, and it's raining - maybe think twice :)