Besides increasing the vocabulary size, one way to use “fewer tokens” for a given task is to adjust how the tokenizer is trained with respect to that task.
If you increase the amount of non-English language representation in your data set, there will be more tokens which cover non-English concepts.
The previous tokenizer infamously required many more tokens to express a given concept in Japanese compared to English. This is likely because the data the tokenizer was trained on (which is not necessarily the same data the GPT model is trained on) had a lot more English data.
Presumably the new tokenizer was trained on data with a higher proportion of foreign language use and lower proportion of non-language use.
