undefined | Better HN

0 pointslittlestymaar1y ago0 comments

GP is talking about context windows, not the number of token used by the tokenizer.

0 comments

sva_1y ago

Somewhat confusingly, it appears the tokenizer vocabulary as well as the context length are both 128k tokens!

Yup, that's why I wanted to clarify things.

j / k navigate · click thread line to collapse

0 pointslittlestymaar1y ago0 comments

GP is talking about context windows, not the number of token used by the tokenizer.

sva_1y ago

Somewhat confusingly, it appears the tokenizer vocabulary as well as the context length are both 128k tokens!

Yup, that's why I wanted to clarify things.

j / k navigate · click thread line to collapse