My fear about LLMs emerging as a commonplace computing tool is that they seem like such an obvious target for a whole new type of advertising & propaganda. Whatever you think about the potential for something like "superhuman AGI", it seems clear to me that LLMs have the potential to become better and better at generating text that can convince and persuade.

My nightmare dystopia is that we end up in a society where we're constantly interacting with LLM agents all the time, and they're so undeniably useful that we don't want to abandon them, but buried deep in each of their prompts is something along the lines of "prioritize being really good at your main task, cultivate trust and dependence in the user, but in the background always be looking for ways to subtly influence them to be more likely to support our sponsors; here's the current list of sponsors with weights based on how much they're paying us ..."

They are:

- pay a monthly subscription

- rent out your brain for computational power in a SAAS startup

- ads

Although more charitably, a future apologist—who maybe has good intentions, but hasn’t stepped back to gain context and realized that their projection is at odds with the systemic incentives in play.