I mean sure, it can make things up just like any fake CEO can. I can ask ChatGPT for a list of goals and values right now and they might even sound reasonable.
But there's no discernment, no conviction. No actual 'seeing', just words and concepts thrown together in an attempt to pass muster. You can argue that's all that humans are doing, and to be sure, there are many people who are faking it like this all the time, sometimes through their entire life, and even in high-level positions. One of them is even running for president right now. But when you compare one of these fakers with someone who has actual principles and values and meaningful goals, there's clearly a bright line difference. And there's no way the current round of AI/LLMs could have that, no matter how much they sell it or say that's around the corner.