undefined | Better HN

0 pointssebzim45003y ago0 comments

>That said, GPT has no model of the world.

I don't think this is necessarily true. Here is an example where researchers trained a transformer to generate legal sequences of moves in the board game Othello. Then they demonstrated that the internal state of the model did, in fact, have a representation of the board.

https://arxiv.org/abs/2210.13382

0 comments

gowld3y ago

That's a GPT and it's specific for one dataset of one game. How would someone extend that to all games and all other fields of human endeavor?

sebzim4500OP3y ago

I'm not sure, the reason you could prove for Othello that the 'world model' exists is that the state is so simple there is really only one reasonable way to represent it with a vector (one component for each square). Even for something like chess there is a huge amount of choice for how to represent the board, yet alone trying represent the state of the actual world.

j / k navigate · click thread line to collapse