This is an important point about GPT-based tools, and it was one of the key parts that Galactica got wrong: it was (over)sold as "an AI scientist", instead of "random crazy thought generator for inspiration/playful ideation assistance".

ChatGPT page every time you open it:

Limitations:

- May occasionally generate incorrect information

- May occasionally produce harmful instructions or biased content

- Limited knowledge of world and events after 2021

Hacker News, on reading this list of caveats:

This looks and feels like an infallible oracle.