undefined | Better HN

0 pointsisaacfung1y ago0 comments

The current gen llms tokenize numbers digit by digit unlike earlier llms.

0 comments

They don't. Which you can easily check with any of the dozen web apps currently implementing the GPT-4o tokenizer.

No, it doesn't help. Bloomberg tried this and it didn't seem to make much difference.

If someone else is interested in the Bloomberg tokenizer:

j / k navigate · click thread line to collapse

They don't. Which you can easily check with any of the dozen web apps currently implementing the GPT-4o tokenizer.

No, it doesn't help. Bloomberg tried this and it didn't seem to make much difference.

If someone else is interested in the Bloomberg tokenizer:

j / k navigate · click thread line to collapse