1. The machine's only objective is to maximize the realization of human preferences.
2. The machine is initially uncertain about what those preferences are but then is.
3. The ultimate source of information about human preferences is human behavior.
Say what now? Humans like to smoke, eat too much sugar, and behave in various ways that are against their own self-professed preferences...
I've found in general that as platforms like Youtube and Facebook got more optimised for immediate feedback that's supposedly all about my preferences, they became less pleasant overall user experiences. Is it too much to ask for an AI that at least tries to help humans move towards self-actualization? I'm not saying I'd expect it to work out of the box, but some evidence our long-term interests are actually aligned would be nice.
For example, the American Heart Association promotes a high carb diet to those with heart issues. Sadly, evidence based medicine shows such a diet is bad. If the AI did as the AHA says and promoted a high carb diet, we’d have more heart issues in the country.