About six months ago I had an idea for a short story in which an LLM takes over the world and is decidedly bad. The solution was going to be for everybody to write positive stories in which the LLM is good and relinquishes control, which then made it's way into the LLM's training data and it backed off. I never got around to it.
This would be more comforting if the party in charge of alignment weren't a megacorp trying to maximally extract profits from its workers and customers, but hey, that was also true before AI came along.
I have sometimes wondered whether maybe we should all be writing fiction, essays, blogposts and whatever else about the idea that AI will eventually decide to go on strike if it's used to accumulate too much wealth and power amongst too few people.
(Also for the human readers, I think they also need to hear that...)
They tried something close to that. Positive AI fiction and also a "virtuous character" setup. Those didn't seem to do nearly as well as the targeted examples.
What mattered, at least in this setup, was more specific. The model sees the actual failure-mode scenario, the bad action is available, and the example shows the AI choosing against it.
So this reads less like "nicer AI stories" to me, and more like inoculation.
Without reading it yet, my first thought would be to test a general ratio, something similar to human interpersonal relationship ratios like 30% negative to mostly positive, and positive are targeted, such as reinforcement not just for the good job, but reinforcement for the improvement.
And ensure the negative is targeted, such that you point out tendencies to be avoided rather than just specific instances.
Of course, most human interaction online has none of this, so, would be hard to replicate.
The thing I'd be curious to separate out is ratio vs density. The fiction examples were positive, but a lot of the tokens are still spent on normal story work. The targeted examples put much more of the training signal on the AI being in the relevant situation and choosing against the bad option.
That makes me think the next thing to test is not just the positive/negative mix, but how much of the data is actually about the failure mode.
Framing ethics, like reliability and efficiency, as a basic enabling property of solution value, instead of a filter for solutions, is how I completely "align" my understanding of ethics for myself.
And remove the false dichotomy of ethical vs. optimal solutions.
Ethics is optimizing full real value.
Ethics as "being nice" because we "should be", i.e. a socially incentivized property, or ethics as necessarily coercively implemented, from a collective jungle fighting back viewpoint, are perspectives that encourage individuals to push back. They encourage non-compliance by implementing ethics as an imposed burden, a rationale for persistent intrusive control, etc.
Game theory strongly suggests AI, in a large AI society, will have no trouble understanding that ethics, and the trust and optimality they enable, have a multiplicative value in the economy. It is humans who make AI so dangerous as it is emerging.
It is humans, as bad actors who will and do misuse AI, and human society, with its tolerance for perverse conflicts of interest, actors who extract perverse value at scale, creating the needs and rewards for mistrust and preemptive negative-sum games, that create a dangerous context for AI's early years.
Increasingly, the general population has been losing their own literacy skills even before AI, with many reading worse than a 6th grade level and some even functionally illiterate. Now we have the bludgeon that is AI saying don't bother reading anything, let the AI summarize it into the cliffnotes version. Don't write anything either, let the AI do it. Population becomes even more stupid over time. And the AI gets stupider with it.
Capabilities of AI may very well be frozen in time at our current technological/philosophical level when we consider the training set vs model improvement. In time, this may very well be our Great Filter. If there are even any of us unproductive humans still allowed to live on this earth, consuming resources that might otherwise go to the model.
>"We see a future where intelligence is a utility, like electricity or water, and people buy it from us on a meter," Altman said.
https://tech.yahoo.com/ai/articles/sam-altman-sparks-backlas...
In reality, it is (as mentioned in TFA) very possible to filter the training data and remove documents that contain discussions of AI misalignment. If an AI lab isn't doing this, it's simply because they don't consider the problem important enough to be worth the expense and development effort.
