This prevalence, and the fact that it's not EVIL doing it but just amoral goal-directed processes, seems to me to be the key to recognizing, fighting back, and fixing society.
We have to figure out some way to fight for our human values, against these optimization processes. I don't think Stross has (or claims to have) a strong answer there... any ideas?
If the only motivator is profit and money, with no feedback loops or sidecars including a factor accounting for human comfort and dignity, you are literally fighting an uphill battle in the space of the objective function.
The answer, I strongly believe, lies in making money somehow account for the human costs associated with conducting business. In whatever economy we operate, if having more money implies you will be treated better than those without, you will see what we are seeing today. If, instead, treating society better means you have more money (which makes life marginally, but not hugely, better than your fellow humans), you will see an incentivization toward social good.
Secular church is definitely one approach, one option to try to fight fire with fire. I've seen compelling analysis tying the increase in intense political tribalization to filling the void left by the general decline in societal religiosity.
Every cause wants to be a cult. (To quote, I think, eyudkowsky)
Previous discussion on video: https://news.ycombinator.com/item?id=16032643
It is no surprise that the event defined by our inability to predict past it did not work out in the way we imagined.
It is definitely an interesting way of looking at things, but I think waters down the "singular" part of the singularity too much.
And this discussion prompts a new thought: What are the military applications of The Singularity? (Don't bother saying that we're all going to live in peace and harmony after it happens. We won't.)
