The difficulty does and has (in the short term) decreased when the amount of hashpower being used goes down.
Presumably if the price went down by a substantial chunk and stayed down for a while, the hashpower would also decrease, and so the difficulty would also decrease.
Also, if electricity prices went up, or if CO2 emissions were taxed, then hashpower would decrease, and the difficulty would go down in turn.