Train’s don’t need to slow down for rough terrain because they can’t handle rough terrain and either route around it or tunnel through it. What they do need to slow down for is the rail network itself.
Poor track conditions, missing block signal systems, and Trains without an automatic cab signal, automatic train stop or automatic train control system "may not exceed 79 mph."
Freight trains really could travel a mostly Class 6 network at 120MPH in the US, we have regulations all the way to class 9. They don’t because that’s not rail is optimized for.